v0.5.114: add Douyu streamers with fan enrichment and live probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -88,9 +88,9 @@ climperor/
|
||||
| `web/fetch_hero_matches.py` | 拉取同英雄近期比赛 + 终局出装/加点 → `web/data/hero_matches.json`(`--source league\|public\|both`;合并后保留最近 N 场胜局,默认 10;天梯需传奇及以上;公开列表过滤 bot/Turbo、仅 ranked lobby;`--public-region china` 优先国服;`--workers` 详情并发 + 凑满即停;可选 `OPENDOTA_API_KEY`;增量补缺失/不足 N 场;`--enrich-item-times` 补购买时间;**仅上分帝 Web**;勿进 recommend) |
|
||||
| `web/fetch_pro_matches.py` | 按明星名单拉近期联赛/锦标赛对局 → `web/data/pro_matches.json`(默认读 `web/data/pro_player_watchlist.json`;`--include-pubs` 另拉天梯 lobby 7;每种 lobby 各保留 `--limit` 场;`--players` 覆盖整文件;`--all-pros` 从 `/proPlayers` 盲抽;含终局出装/加点;进 `refresh_web` daily 且带 `--include-pubs`;**仅上分帝 Web「比赛」**;勿进 recommend) |
|
||||
| `web/fetch_leaderboards.py` | 拉取 Valve Immortal 四区榜 Top100 → `web/data/leaderboards.json`(**仅上分帝 Web「排行」**;无 MMR/account_id;勿进 recommend) |
|
||||
| `web/fetch_streamers.py` | 从抖音主页补全 `web/data/streamers.json` 的昵称/签名/关注粉丝获赞/头像(手工名单 + 直播间/主页 URL;支持 `v.douyin.com` 短链;失败保留旧值;**不**探测开播(由 `fetch_streamer_live.py` 负责);**仅上分帝 Web「主播」**;勿进 recommend;进 `refresh_web` daily) |
|
||||
| `web/fetch_streamer_live.py` | 探测主播真实在播状态回写 `web/data/streamers.json` 的 `is_live`/`live_probed_at`:抖音解析直播间 SSR 页 `roomStore.roomInfo.room.status`(2 在播 / 4 下播;预热 cookie + ~1s 间隔;web_rid 校验),B 站走 `Room/get_info`(`live_status==1` 在播,轮播算下播);探测失败清为 `is_live:false` 并去掉 `live_probed_at`(与 `/api/live-status` 一致,不沿用旧直播中)、始终 exit 0;`--ids a,b` 限范围、`--dry-run` 只打印;仅 Web;进 `refresh_web` daily(角标以访问触发的 live API 为准,daily 仅作 data.json 兜底) |
|
||||
| `web/frontend/functions/api/live-status.js` | Pages Function `GET /api/live-status`:访问触发的在播探测(逻辑同 `fetch_streamer_live.py`),读 `data.json` 的 `streamers.streamers`;Cache API 固定键 + isolate 内 in-flight 合并(5 分钟新鲜窗口,**无 KV**);抖音从数据中心 IP 失败属预期 → 失败主播一律 `is_live:false` + `stale:true`(**不**沿用旧的直播中);全失败回 `stale-override` 空角标表或 `error`,永不 500;导出时拷贝 `functions/`;**部署须 `cwd=dist` 跑 wrangler**(Functions 相对 cwd 解析) |
|
||||
| `web/fetch_streamers.py` | 从抖音 / 斗鱼主页补全 `web/data/streamers.json` 的昵称/签名/关注粉丝获赞(播放)/头像(手工名单 + 直播间/主页 URL;抖音支持 `v.douyin.com` 短链;斗鱼优先 `v.douyu.com/author/<hash>`(兼容 `author-video`)的 `$DATA`,仅房间号时从直播间 HTML 解析 `up_id` 再拉作者页,失败才回退 `betard`;失败保留旧值;**不**探测开播(由 `fetch_streamer_live.py` 负责);**仅上分帝 Web「主播」**;勿进 recommend;进 `refresh_web` daily) |
|
||||
| `web/fetch_streamer_live.py` | 探测主播真实在播状态回写 `web/data/streamers.json` 的 `is_live`/`live_probed_at`:抖音解析直播间 SSR 页 `roomStore.roomInfo.room.status`(2 在播 / 4 下播;预热 cookie + ~1s 间隔;web_rid 校验),B 站走 `Room/get_info`(`live_status==1` 在播,轮播算下播),斗鱼走 `betard/{room_id}`(`show_status==1` 在播,`videoLoop==1` 轮播算下播);探测失败清为 `is_live:false` 并去掉 `live_probed_at`(与 `/api/live-status` 一致,不沿用旧直播中)、始终 exit 0;`--ids a,b` 限范围、`--dry-run` 只打印;仅 Web;进 `refresh_web` daily(角标以访问触发的 live API 为准,daily 仅作 data.json 兜底) |
|
||||
| `web/frontend/functions/api/live-status.js` | Pages Function `GET /api/live-status`:访问触发的在播探测(逻辑同 `fetch_streamer_live.py`,含抖音 / B 站 / 斗鱼),读 `data.json` 的 `streamers.streamers`;Cache API 固定键 + isolate 内 in-flight 合并(5 分钟新鲜窗口,**无 KV**);抖音从数据中心 IP 失败属预期 → 失败主播一律 `is_live:false` + `stale:true`(**不**沿用旧的直播中);全失败回 `stale-override` 空角标表或 `error`,永不 500;导出时拷贝 `functions/`;**部署须 `cwd=dist` 跑 wrangler**(Functions 相对 cwd 解析) |
|
||||
| `web/frontend/functions/api/mobile-demand.js` | Pages Function `GET\|POST /api/mobile-demand`:移动端「催更」需求计数(Cache API 存 `count`,**无 KV**;边缘竞态/驱逐可能少计或重置);本机 `serve_relations.py` 写 `web/.refresh/mobile_demand.json`;前端 `mobile-gate.js` 用 UA 识别手机/平板并拦截,`localStorage` 同设备只 POST 一次 |
|
||||
| `web/frontend/mobile-gate.js` | 移动端门禁(`<head>` 早载):`html.mobile-client` + 催更按钮;设 `window.__CLIMPEROR_MOBILE__` 供 `app.js` 跳过桌面 boot |
|
||||
| `web/fetch_item_shop.py` | 官网商店 11 列目录(dota2.com.cn/itemscategory)+ 合成图 → `web/data/item_shop.json` + 图标 |
|
||||
@@ -109,7 +109,7 @@ climperor/
|
||||
| `web/data/pro_player_watchlist.json` | 明星选手 OpenDota `account_id` 手工名单(对照 Dotabuff / Liquipedia;含现役战队席位与昔日国服明星;外号:查理斯→Chalice,Somnus 即 Maybe;CN/EU/SA 等);`fetch_pro_matches.py` 默认只拉此名单;队名可过期,以 id 为准;**不**进 recommend |
|
||||
| `web/data/pro_matches.json` | 明星选手近期联赛/锦标赛(可选天梯)对局(`by_pro`/`by_hero` + 终局出装/加点 + `lobby_type`/`origin`;由 `fetch_pro_matches.py` 生成;Web 顶层「比赛」侧栏筛「全部 / 职业 / 国服」+ 选手,与英雄详情「近期比赛」合并展示;进 daily;**不**参与局内推荐) |
|
||||
| `web/data/leaderboards.json` | Valve Immortal 四区 Top100(`china`/`europe`/`americas`/`se_asia`;仅排名+昵称等;由 `fetch_leaderboards.py` 生成;Web「排行」选手榜只读;**不**参与局内推荐) |
|
||||
| `web/data/streamers.json` | 主播目录(手工 `live_url`/profile URL + 常用英雄 + 可选精选视频 `video`/`video_title` + 可选 `video_poster`/`video_fit`/`video_crop`/`video_aspect`;抖音/B 站 profile 补全;`platform_meta`;Web 卡片:头像行 = 头像 \| 昵称+账号/获赞粉丝 \|「关注」,签名(`signature`)独立全宽行(最长 3 行);在播时粉环 +「直播」角标叠在环底(抖音式 `bottom:-6px`,无间距);列表排序:先 `is_live` 再粉丝数降序;视口分档滚播(远处不拉、近处 metadata、中部 `canplay` 且单路 `preload=auto`)+ 同名 JPG 封面占位;有 `live_url` 时点头像进直播间;**「直播」角标只信本轮成功探测**(线上 `/api/live-status` 边缘缓存 5 分钟;本地 `serve_relations` 同逻辑缓存 60s;`stale`/失败不显示角标;`data.json` daily 仅作首屏兜底直至接口返回;`live_url` 仅作点击入口);由 `fetch_streamers.py` 补全;Web「主播」只读;**不**参与局内推荐) |
|
||||
| `web/data/streamers.json` | 主播目录(手工 `live_url`/profile URL + 常用英雄 + 可选精选视频 `video`/`video_title` + 可选 `video_poster`/`video_fit`/`video_crop`/`video_aspect`;抖音 profile 补全;平台含抖音 / B 站 / 斗鱼;`platform_meta`;Web 卡片:头像行 = 头像 \| 昵称+账号/获赞粉丝 \|「关注」,签名(`signature`)独立全宽行(最长 3 行);在播时粉环 +「直播」角标叠在环底(抖音式 `bottom:-6px`,无间距);列表排序:先 `is_live` 再粉丝数降序;视口分档滚播(远处不拉、近处 metadata、中部 `canplay` 且单路 `preload=auto`)+ 同名 JPG 封面占位;有 `live_url` 时点头像进直播间;**「直播」角标只信本轮成功探测**(线上 `/api/live-status` 边缘缓存 5 分钟;本地 `serve_relations` 同逻辑缓存 60s;`stale`/失败不显示角标;`data.json` daily 仅作首屏兜底直至接口返回;`live_url` 仅作点击入口);由 `fetch_streamers.py` 补全;Web「主播」只读;**不**参与局内推荐) |
|
||||
| `web/data/stratz_hero_meta.json` | STRATZ 各勋章段位近 N 周 pick/win(`weeks`/`latest`)+ **同段位最近 1 周分路**(`winWeek`+`positionIds`)+ `meta_board`(由 `fetch_stratz_meta.py` 生成;Web 英雄详情「走势」优先 + 顶层「走势」`#/trends` 近 N 周榜;**不**参与局内推荐) |
|
||||
| `web/data/stratz_matchup_tops.json` | STRATZ 对位/协同 Top(counters/countered/synergies;由 `fetch_stratz_meta.py` 生成;全局聚合 + `scope`/`stale`/`fetched_at`;Web「对位」Tab;与定性 `relations.json` 分开展示;**不**参与局内推荐) |
|
||||
| `web/data/item_shop.json` | 商店 11 列目录(官网 basic/upgrade;由 `fetch_item_shop.py` 生成;物品页只读) |
|
||||
@@ -163,7 +163,7 @@ Dota 2 GSI → pc/gsi_watch.py (:3223)
|
||||
|
||||
- **合规**:仅 GSI + 屏幕截图;**禁止**读进程内存、注入、绕过 VAC。
|
||||
- **GSI 范围**:普通玩家视角拿不到双方 pick;GSI 只作阶段触发、`team_slot`、本人 `hero`、本机 `accountid`/`steamid`。本人顶栏槽位**只信** GSI,不用截屏名字亮度猜测。
|
||||
- **克制 / 搭档数据**:机制克制/搭档只用定性边存 `shared/data/relations.json`(克制有向 + 搭档无向 + 理由),**不要**用胜率/场次表达机制克制,也**不要**写入 `shared/data/heroes.json`。Web 英雄页三视图:克制 / 被克制 / 搭档。OpenDota 段位胜率/场次单独存 `web/data/hero_stats.json`(近约 7 天窗;上场率 = 出场/(Σ出场/10);冠绝样本过小时与超凡合并),**仅**上分帝 Web 英雄详情「走势」**无 STRATZ 时兜底**。STRATZ 周胜率/分路/`meta_board` 存 `web/data/stratz_hero_meta.json`(三卡与分路均取 **当前选中勋章 · 最近 1 周** `winWeek`;近 8 周列表从新到旧;分路在 8 周列表上方),对位 Top 存 `web/data/stratz_matchup_tops.json`(英雄详情「走势」与顶层「走势」榜 `#/trends` 优先用 STRATZ;「对位」Tab 只读),**禁止**合并进 relations/heroes,禁止 `pc/recommend.py` 读取。同英雄近期比赛出装/加点单独存 `web/data/hero_matches.json`,**仅** Web「近期比赛」Tab,禁止进 recommend。Valve Immortal 四区榜单独存 `web/data/leaderboards.json`,**仅** Web「排行」页选手榜,禁止进 recommend。主播目录单独存 `web/data/streamers.json`(手工名单 + 抖音 profile 补全;首版仅抖音),**仅** Web 顶层「主播」与英雄详情「主播」Tab,禁止进 recommend;由 `refresh_web` daily 软失败刷新粉丝等字段(不阻断整档)。
|
||||
- **克制 / 搭档数据**:机制克制/搭档只用定性边存 `shared/data/relations.json`(克制有向 + 搭档无向 + 理由),**不要**用胜率/场次表达机制克制,也**不要**写入 `shared/data/heroes.json`。Web 英雄页三视图:克制 / 被克制 / 搭档。OpenDota 段位胜率/场次单独存 `web/data/hero_stats.json`(近约 7 天窗;上场率 = 出场/(Σ出场/10);冠绝样本过小时与超凡合并),**仅**上分帝 Web 英雄详情「走势」**无 STRATZ 时兜底**。STRATZ 周胜率/分路/`meta_board` 存 `web/data/stratz_hero_meta.json`(三卡与分路均取 **当前选中勋章 · 最近 1 周** `winWeek`;近 8 周列表从新到旧;分路在 8 周列表上方),对位 Top 存 `web/data/stratz_matchup_tops.json`(英雄详情「走势」与顶层「走势」榜 `#/trends` 优先用 STRATZ;「对位」Tab 只读),**禁止**合并进 relations/heroes,禁止 `pc/recommend.py` 读取。同英雄近期比赛出装/加点单独存 `web/data/hero_matches.json`,**仅** Web「近期比赛」Tab,禁止进 recommend。Valve Immortal 四区榜单独存 `web/data/leaderboards.json`,**仅** Web「排行」页选手榜,禁止进 recommend。主播目录单独存 `web/data/streamers.json`(手工名单 + 抖音 profile 补全;平台含抖音 / B 站 / 斗鱼),**仅** Web 顶层「主播」与英雄详情「主播」Tab,禁止进 recommend;由 `refresh_web` daily 软失败刷新粉丝等字段(不阻断整档)。
|
||||
- **装备机制 / 怕的装备**:标签与技能汇总来自 Valve/OpenDota 自动抽取 + `item_tag_overrides.json`;`item_fears.py` 规则映射到英雄弱点。`fetch_item_counter_stats.py` 的对阵购买率/胜率差仅作为观测证据:须减去同装备全局基线,对全部英雄的已有机制候选只允许小幅调序;统计发现的新组合即使同时满足 `games≥100`、购买率提升 `≥3pp`、条件胜率差 `≥1.5pp`、两项双比例检验 `z≥1.96`,仍须确认机制成立后手工写入 overrides,禁止直接把高相关当因果克制。大段技能文案只进 `web/data/hero_abilities.json` / `items_meta.json`,**不要**塞进 `heroes.json`。本阶段仅上分帝 Web 展示,**不对局内出装推荐**。核心装「使用率」为 `hero_items` 列表内相对热度归一化,非绝对出场率、无段位维度。技能/物品「施加」类 tags(机制查询页)与技能 `dispellable`(效果能否被驱散)严格区分;**禁止**写入 recommend / relations。
|
||||
- **机制查询页**:顶层 `#/mechanics[/{effect}]`;侧栏弱/强驱散 + 核心控制;结果为施加该效果的技能与物品。数据边界:仅 Web。
|
||||
- **模板策略**:只维护 CDN 层。皮肤问题用会话策略(选人可改判、决策 `allow_revise=False`、best 帧偏 HERO_SELECTION),不要为皮肤加模板库,也不要复活 real 双层库。
|
||||
|
||||
@@ -4,6 +4,13 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.5.114] - 2026-07-30
|
||||
|
||||
### Added
|
||||
|
||||
- 主播资料补全支持斗鱼:`v.douyu.com/author/<hash>`(兼容 `author-video`)拉粉丝/关注/播放/头像;仅有房间号时从直播间 HTML 解析 `up_id` 再拉作者页(失败才回退 `betard`)。斗鱼卡片展示「关注 / 播放 / 粉丝」。
|
||||
- 主播页收录斗鱼「踏上征途167(狗皇)」「天残少年K9」「叶子长青K」、抖音「浙大小鱼王—dota2」与 B 站「Dota2绝中绝」;开播探测支持斗鱼(`betard`)。
|
||||
|
||||
## [0.5.112] - 2026-07-30
|
||||
|
||||
### Changed
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 34 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 9.9 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 12 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 43 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.1 KiB |
+102
-2
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"fetched_at": "2026-07-28T14:20:00+00:00",
|
||||
"source": "manual+douyin+bilibili",
|
||||
"fetched_at": "2026-07-29T18:37:21.491922+00:00",
|
||||
"source": "manual+douyin+bilibili+douyu",
|
||||
"platform_meta": {
|
||||
"douyin": {
|
||||
"label_zh": "抖音",
|
||||
@@ -9,6 +9,10 @@
|
||||
"bilibili": {
|
||||
"label_zh": "哔哩哔哩",
|
||||
"icon": "ui-icon/platform_bilibili.png"
|
||||
},
|
||||
"douyu": {
|
||||
"label_zh": "斗鱼",
|
||||
"icon": "ui-icon/platform_douyu.png"
|
||||
}
|
||||
},
|
||||
"streamers": [
|
||||
@@ -480,6 +484,102 @@
|
||||
"profile_fetched_at": "2026-07-29T09:54:00+00:00",
|
||||
"is_live": true,
|
||||
"live_probed_at": "2026-07-29T09:54:00+00:00"
|
||||
},
|
||||
{
|
||||
"id": "gouhuang",
|
||||
"platform": "douyu",
|
||||
"live_url": "https://www.douyu.com/957090",
|
||||
"profile_url": "https://v.douyu.com/author/EqAvg1lQD75L",
|
||||
"tagline": "米波绝活 · 狗皇",
|
||||
"heroes": [
|
||||
"meepo",
|
||||
"broodmother",
|
||||
"alchemist"
|
||||
],
|
||||
"nickname": "踏上征途167",
|
||||
"unique_id": "957090",
|
||||
"signature": "娱乐DOTA 轻松氛围",
|
||||
"avatar": "streamer_avatars/gouhuang.jpg",
|
||||
"is_live": false,
|
||||
"live_probed_at": "2026-07-29T18:16:27.171378+00:00",
|
||||
"profile_fetched_at": "2026-07-29T18:37:18.228078+00:00",
|
||||
"following_count": 868,
|
||||
"follower_count": 280201,
|
||||
"total_favorited": 9673470
|
||||
},
|
||||
{
|
||||
"id": "juezhongjue",
|
||||
"platform": "bilibili",
|
||||
"live_url": "https://live.bilibili.com/1747114599",
|
||||
"profile_url": "https://space.bilibili.com/520653835",
|
||||
"tagline": "绝活打野炼金",
|
||||
"heroes": [
|
||||
"alchemist"
|
||||
],
|
||||
"nickname": "Dota2绝中绝",
|
||||
"unique_id": "520653835",
|
||||
"signature": "绝活打野炼金。鄙人不擅长对线对线,但极其擅长与野怪掰头!",
|
||||
"follower_count": 433,
|
||||
"avatar": "streamer_avatars/juezhongjue.jpg",
|
||||
"is_live": false,
|
||||
"live_probed_at": "2026-07-29T18:08:20.096701+00:00"
|
||||
},
|
||||
{
|
||||
"id": "k9",
|
||||
"platform": "douyu",
|
||||
"live_url": "https://www.douyu.com/235520",
|
||||
"profile_url": "https://v.douyu.com/author/JPw9YOLKlw5X",
|
||||
"tagline": "绝活卡尔",
|
||||
"heroes": [
|
||||
"invoker"
|
||||
],
|
||||
"nickname": "18yearsold天残少年K9",
|
||||
"unique_id": "235520",
|
||||
"signature": "k9:午夜牢车教室!.",
|
||||
"avatar": "streamer_avatars/k9.jpg",
|
||||
"is_live": true,
|
||||
"live_probed_at": "2026-07-29T18:16:27.995333+00:00",
|
||||
"following_count": 16,
|
||||
"follower_count": 465781,
|
||||
"total_favorited": 26074078,
|
||||
"profile_fetched_at": "2026-07-29T18:23:59.723287+00:00"
|
||||
},
|
||||
{
|
||||
"id": "yezi",
|
||||
"platform": "douyu",
|
||||
"live_url": "https://www.douyu.com/246195",
|
||||
"profile_url": "https://v.douyu.com/author/Kqy70jkrpdXG",
|
||||
"tagline": "绝活火猫",
|
||||
"heroes": [
|
||||
"ember_spirit"
|
||||
],
|
||||
"nickname": "叶子长青K",
|
||||
"unique_id": "246195",
|
||||
"signature": "QQ:394867894 网易云搜用户 淡然听听歌",
|
||||
"avatar": "streamer_avatars/yezi.jpg",
|
||||
"is_live": false,
|
||||
"live_probed_at": "2026-07-29T18:16:28.416689+00:00",
|
||||
"profile_fetched_at": "2026-07-29T18:37:20.691635+00:00",
|
||||
"following_count": 134,
|
||||
"follower_count": 286391,
|
||||
"total_favorited": 6336746
|
||||
},
|
||||
{
|
||||
"id": "zhedaxiaoyuwang",
|
||||
"platform": "douyin",
|
||||
"profile_url": "https://v.douyin.com/tSe_vsZ8r7o/",
|
||||
"tagline": "小鱼人",
|
||||
"heroes": [
|
||||
"slark"
|
||||
],
|
||||
"nickname": "浙大小鱼王—dota2",
|
||||
"unique_id": "89748079578",
|
||||
"signature": "本硕985,浙大在读 9700冲万分",
|
||||
"following_count": 52,
|
||||
"follower_count": 1802,
|
||||
"total_favorited": 842,
|
||||
"avatar": "streamer_avatars/zhedaxiaoyuwang.jpg",
|
||||
"profile_fetched_at": "2026-07-29T18:26:00+00:00"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -52,7 +52,7 @@ from shared.paths import (
|
||||
|
||||
from serve_relations import WEB_DIR, build_payload
|
||||
|
||||
SITE_VERSION = "0.5.112"
|
||||
SITE_VERSION = "0.5.114"
|
||||
DEFAULT_OSS_BASE = "https://climperor.oss-cn-shanghai.aliyuncs.com"
|
||||
|
||||
|
||||
|
||||
@@ -13,6 +13,9 @@ Approach (verified 2026-07):
|
||||
- Bilibili: public API ``api.live.bilibili.com/room/v1/Room/get_info`` with the
|
||||
numeric room id taken from the ``live_url`` path; ``data.live_status == 1``
|
||||
means live (0 offline, 2 replay — replay is treated as offline). No login.
|
||||
- Douyu: public ``www.douyu.com/betard/{room_id}``; ``room.show_status == 1``
|
||||
means live, ``== 2`` offline; ``room.videoLoop == 1`` (carousel) counts as
|
||||
offline.
|
||||
- Douyin: one shared cookie session is warmed up (www.douyin.com +
|
||||
live.douyin.com), then per room we GET ``live.douyin.com/{web_rid}`` with a
|
||||
browser UA (the numeric ``live_url`` path segment is the ``web_rid``). The
|
||||
@@ -65,10 +68,12 @@ TIMEOUT = 20
|
||||
# Douyin rate-limits aggressively; keep ~1s spacing between its requests.
|
||||
DOUYIN_SPACING = 1.0
|
||||
BILIBILI_SPACING = 0.5
|
||||
DOUYU_SPACING = 0.3
|
||||
|
||||
DOUYIN_HOME = "https://www.douyin.com/"
|
||||
DOUYIN_LIVE_HOME = "https://live.douyin.com/"
|
||||
BILIBILI_INFO_URL = "https://api.live.bilibili.com/room/v1/Room/get_info?room_id={room_id}"
|
||||
DOUYU_BETARD_URL = "https://www.douyu.com/betard/{room_id}"
|
||||
|
||||
# Escaped JSON inside the SSR pace chunks: \"roomStore\":{\"roomInfo\":{\"room\":{
|
||||
DOUYIN_ROOMSTORE_RE = re.compile(
|
||||
@@ -152,8 +157,27 @@ def probe_bilibili(room_id: str) -> bool:
|
||||
return data.get("live_status") == 1
|
||||
|
||||
|
||||
def probe_douyu(room_id: str) -> bool:
|
||||
"""show_status: 1 live, 2 offline; videoLoop carousel counts as offline."""
|
||||
payload = http_utils.http_json(
|
||||
DOUYU_BETARD_URL.format(room_id=room_id), timeout=TIMEOUT
|
||||
)
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("douyu betard returned non-object")
|
||||
room = payload.get("room")
|
||||
if not isinstance(room, dict):
|
||||
raise ValueError("douyu betard returned no room")
|
||||
if int(room.get("videoLoop") or 0) == 1:
|
||||
return False
|
||||
status = room.get("show_status")
|
||||
try:
|
||||
return int(status) == 1
|
||||
except (TypeError, ValueError) as e:
|
||||
raise ValueError(f"douyu unexpected show_status {status!r}") from e
|
||||
|
||||
|
||||
def room_ref_from_url(live_url: str) -> str | None:
|
||||
"""First path segment of the live room URL (douyin web_rid / bilibili room id)."""
|
||||
"""First path segment of the live room URL (douyin/bilibili/douyu room id)."""
|
||||
path = urllib.parse.urlparse(live_url.strip()).path.strip("/")
|
||||
if not path:
|
||||
return None
|
||||
@@ -167,6 +191,8 @@ def live_platform_from_url(live_url: str, fallback: str = "") -> str:
|
||||
return "bilibili"
|
||||
if "douyin.com" in host:
|
||||
return "douyin"
|
||||
if "douyu.com" in host:
|
||||
return "douyu"
|
||||
return (fallback or "").strip().lower()
|
||||
|
||||
|
||||
@@ -210,6 +236,9 @@ def probe_streamers(
|
||||
elif platform == "bilibili":
|
||||
is_live = probe_bilibili(ref)
|
||||
time.sleep(BILIBILI_SPACING)
|
||||
elif platform == "douyu":
|
||||
is_live = probe_douyu(ref)
|
||||
time.sleep(DOUYU_SPACING)
|
||||
else:
|
||||
print(f"skip {sid}: platform={platform!r} unsupported", flush=True)
|
||||
continue
|
||||
|
||||
+217
-16
@@ -1,16 +1,23 @@
|
||||
"""Fetch Douyin profile fields into data/streamers.json.
|
||||
"""Fetch Douyin / Douyu profile fields into data/streamers.json.
|
||||
|
||||
Manual seed fields (id / platform / live_url / profile_url / heroes / tagline)
|
||||
are preserved. Profile enrichment (nickname, signature, counts, avatar) is
|
||||
best-effort via Douyin HTML RENDER_DATA + text fallback; failures keep the
|
||||
previous values.
|
||||
best-effort:
|
||||
|
||||
- Douyin: HTML RENDER_DATA + text fallback
|
||||
- Douyu: ``v.douyu.com/author/<hash>`` (or ``author-video/<hash>``) page
|
||||
``window.$DATA`` (fans / following / plays / avatar). Room-only rows
|
||||
resolve ``up_id`` from the live-room HTML then fetch the author page;
|
||||
if that fails, fall back to ``betard`` (nickname / avatar / bio, no fans).
|
||||
|
||||
Failures keep the previous values.
|
||||
|
||||
Preview only — do not merge into relations/heroes or recommend.
|
||||
Part of refresh_web ``daily`` (soft-fail: never aborts the tier).
|
||||
|
||||
Usage:
|
||||
python fetch_streamers.py
|
||||
python fetch_streamers.py --ids xiaowang
|
||||
python fetch_streamers.py --ids xiaowang k9
|
||||
python fetch_streamers.py --out data/streamers.json
|
||||
"""
|
||||
|
||||
@@ -32,8 +39,8 @@ import urllib.request
|
||||
from datetime import datetime, timezone
|
||||
from typing import Any
|
||||
|
||||
from shared.http_utils import write_json_atomic
|
||||
from shared.paths import DATA, STREAMER_AVATARS
|
||||
from shared.http_utils import http_json, write_json_atomic
|
||||
from shared.paths import DATA, ROOT, STREAMER_AVATARS
|
||||
|
||||
OUT = DATA / "streamers.json"
|
||||
AVATAR_DIR = STREAMER_AVATARS
|
||||
@@ -58,6 +65,17 @@ HTML_AVATAR_RE = re.compile(
|
||||
re.IGNORECASE,
|
||||
)
|
||||
SEC_UID_RE = re.compile(r"/user/(MS4wLjABAAAA[A-Za-z0-9_-]+)")
|
||||
DOUYU_AUTHOR_HASH_RE = re.compile(
|
||||
r"(?:v\.)?douyu\.com/author(?:-video)?/([A-Za-z0-9]+)", re.IGNORECASE
|
||||
)
|
||||
DOUYU_ROOM_RE = re.compile(
|
||||
r"(?:www\.)?douyu\.com/(\d+)(?:/|$|\?)", re.IGNORECASE
|
||||
)
|
||||
DOUYU_DATA_RE = re.compile(r"window\.\$DATA=(\{.*?\}),\$", re.DOTALL)
|
||||
DOUYU_BARE_KEY_RE = re.compile(r"([{\s,])([A-Za-z_][A-Za-z0-9_]*)\s*:")
|
||||
# Room HTML embeds up_id in plain JSON and/or JSON-escaped script strings.
|
||||
DOUYU_UP_ID_RE = re.compile(r'\\?"up_id\\?"\s*:\s*\\?"([A-Za-z0-9]+)\\?"')
|
||||
DOUYU_BETARD_URL = "https://www.douyu.com/betard/{room_id}"
|
||||
# Fields fetch may overwrite; manual seed keys are never removed.
|
||||
PROFILE_KEYS = (
|
||||
"nickname",
|
||||
@@ -104,7 +122,11 @@ def _opener() -> urllib.request.OpenerDirector:
|
||||
|
||||
|
||||
def _get(
|
||||
opener: urllib.request.OpenerDirector, url: str, *, timeout: int = 30
|
||||
opener: urllib.request.OpenerDirector,
|
||||
url: str,
|
||||
*,
|
||||
timeout: int = 30,
|
||||
referer: str = "https://www.douyin.com/",
|
||||
) -> tuple[str, str]:
|
||||
"""Return (final_url, html)."""
|
||||
req = urllib.request.Request(
|
||||
@@ -113,7 +135,7 @@ def _get(
|
||||
"User-Agent": BROWSER_UA,
|
||||
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
|
||||
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
|
||||
"Referer": "https://www.douyin.com/",
|
||||
"Referer": referer,
|
||||
},
|
||||
)
|
||||
with opener.open(req, timeout=timeout) as resp:
|
||||
@@ -306,7 +328,7 @@ def resolve_profile_url(
|
||||
return url, None
|
||||
|
||||
|
||||
def download_avatar(url: str, dest: Path) -> bool:
|
||||
def download_avatar(url: str, dest: Path, *, referer: str | None = None) -> bool:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
# Prefer a larger CDN variant when the URL embeds a size token.
|
||||
candidates = [url]
|
||||
@@ -314,12 +336,22 @@ def download_avatar(url: str, dest: Path) -> bool:
|
||||
candidates.insert(0, url.replace("/100x100/", "/720x720/"))
|
||||
if "300x300" in url:
|
||||
candidates.insert(0, url.replace("300x300", "720x720"))
|
||||
if "_avatar_middle." in url:
|
||||
candidates.insert(0, url.replace("_avatar_middle.", "_avatar_big."))
|
||||
if "_middle.jpg" in url:
|
||||
candidates.insert(0, url.replace("_middle.jpg", "_big.jpg"))
|
||||
host = urllib.parse.urlparse(url).netloc.lower()
|
||||
if referer is None:
|
||||
if "douyu" in host:
|
||||
referer = "https://www.douyu.com/"
|
||||
else:
|
||||
referer = "https://www.douyin.com/"
|
||||
for candidate in candidates:
|
||||
req = urllib.request.Request(
|
||||
candidate,
|
||||
headers={
|
||||
"User-Agent": BROWSER_UA,
|
||||
"Referer": "https://www.douyin.com/",
|
||||
"Referer": referer,
|
||||
},
|
||||
)
|
||||
try:
|
||||
@@ -336,6 +368,153 @@ def download_avatar(url: str, dest: Path) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def parse_douyu_dollar_data(blob: str) -> dict:
|
||||
"""Parse Douyu ``window.$DATA={...}`` JS object (bare keys) into a dict."""
|
||||
quoted = DOUYU_BARE_KEY_RE.sub(r'\1"\2":', blob.strip())
|
||||
data = json.loads(quoted)
|
||||
if not isinstance(data, dict):
|
||||
raise ValueError("douyu $DATA is not an object")
|
||||
return data
|
||||
|
||||
|
||||
def extract_profile_from_douyu_data(data: dict) -> dict:
|
||||
"""Map Douyu author ``$DATA`` fields onto streamer profile keys."""
|
||||
out: dict[str, Any] = {}
|
||||
nick = _first_str(data.get("name"), data.get("nickname"))
|
||||
if nick:
|
||||
out["nickname"] = nick
|
||||
room_id = _first_str(data.get("roomId"), data.get("room_id"))
|
||||
if room_id:
|
||||
out["unique_id"] = room_id
|
||||
out["live_url"] = f"https://www.douyu.com/{room_id}"
|
||||
up_id = _first_str(data.get("upId"), data.get("up_id"))
|
||||
if up_id:
|
||||
# Canonical homepage (author space); author-video is an alias.
|
||||
out["profile_url"] = f"https://v.douyu.com/author/{up_id}"
|
||||
# Author bio only — do not fall back to live room title (would wipe
|
||||
# hand-seeded / betard signatures on every refresh).
|
||||
contents = _first_str(data.get("contents"), data.get("ownerAuthContents"))
|
||||
if contents:
|
||||
out["signature"] = contents
|
||||
out["following_count"] = _first_int(data.get("upFollowNum"), data.get("up_follow_num"))
|
||||
out["follower_count"] = _first_int(data.get("subscribeNum"), data.get("subscribe_num"))
|
||||
# Video play count — shown as「播放」on Douyu cards.
|
||||
out["total_favorited"] = _first_int(data.get("playCount"), data.get("play_count"))
|
||||
avatar = _first_str(data.get("avatar"))
|
||||
if avatar:
|
||||
out["avatar_url"] = avatar.replace(r"\/", "/")
|
||||
return out
|
||||
|
||||
|
||||
def fetch_douyu_author_profile(
|
||||
opener: urllib.request.OpenerDirector, profile_url: str
|
||||
) -> dict:
|
||||
"""Fetch Douyu author / author-video page and parse ``window.$DATA``."""
|
||||
m = DOUYU_AUTHOR_HASH_RE.search(profile_url)
|
||||
if not m:
|
||||
raise ValueError(f"not a douyu author url: {profile_url!r}")
|
||||
hash_id = m.group(1)
|
||||
# Prefer /author/; fall back to /author-video/ if $DATA is missing.
|
||||
errors: list[str] = []
|
||||
for path in (f"author/{hash_id}", f"author-video/{hash_id}"):
|
||||
page_url = f"https://v.douyu.com/{path}"
|
||||
try:
|
||||
_, html = _get(opener, page_url, referer="https://v.douyu.com/")
|
||||
except (urllib.error.URLError, TimeoutError, OSError) as e:
|
||||
errors.append(f"{path}: {e}")
|
||||
continue
|
||||
data_m = DOUYU_DATA_RE.search(html)
|
||||
if not data_m:
|
||||
errors.append(f"{path}: no window.$DATA")
|
||||
continue
|
||||
profile = extract_profile_from_douyu_data(
|
||||
parse_douyu_dollar_data(data_m.group(1))
|
||||
)
|
||||
if not any(
|
||||
profile.get(k) is not None
|
||||
for k in ("nickname", "follower_count", "avatar_url", "unique_id")
|
||||
):
|
||||
errors.append(f"{path}: parsed empty")
|
||||
continue
|
||||
return profile
|
||||
raise ValueError("; ".join(errors) or "douyu author page failed")
|
||||
|
||||
|
||||
def extract_douyu_up_id_from_room_html(html: str) -> str | None:
|
||||
"""Pull author ``up_id`` hash embedded in the live-room page HTML."""
|
||||
m = DOUYU_UP_ID_RE.search(html or "")
|
||||
return m.group(1) if m else None
|
||||
|
||||
|
||||
def resolve_douyu_author_url_from_room(
|
||||
opener: urllib.request.OpenerDirector, room_id: str
|
||||
) -> str:
|
||||
"""Fetch ``www.douyu.com/<rid>`` and build the author homepage URL."""
|
||||
page_url = f"https://www.douyu.com/{room_id}"
|
||||
_, html = _get(opener, page_url, referer="https://www.douyu.com/")
|
||||
up_id = extract_douyu_up_id_from_room_html(html)
|
||||
if not up_id:
|
||||
raise ValueError(f"douyu room {room_id}: no up_id in HTML")
|
||||
return f"https://v.douyu.com/author/{up_id}"
|
||||
|
||||
|
||||
def fetch_douyu_room_profile(room_id: str) -> dict:
|
||||
"""Fallback enrichment from live-room ``betard`` (no fan counts)."""
|
||||
payload = http_json(DOUYU_BETARD_URL.format(room_id=room_id), timeout=20)
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError("douyu betard returned non-object")
|
||||
room = payload.get("room")
|
||||
if not isinstance(room, dict):
|
||||
raise ValueError("douyu betard returned no room")
|
||||
out: dict[str, Any] = {
|
||||
"unique_id": str(room_id),
|
||||
"live_url": f"https://www.douyu.com/{room_id}",
|
||||
}
|
||||
nick = _first_str(room.get("nickname"), room.get("owner_name"))
|
||||
if nick:
|
||||
out["nickname"] = nick
|
||||
details = _first_str(room.get("show_details"), room.get("room_name"))
|
||||
if details:
|
||||
out["signature"] = details
|
||||
avatar = room.get("avatar")
|
||||
avatar_url = None
|
||||
if isinstance(avatar, dict):
|
||||
avatar_url = _first_str(avatar.get("big"), avatar.get("middle"), avatar.get("small"))
|
||||
elif isinstance(avatar, str):
|
||||
avatar_url = avatar
|
||||
if not avatar_url:
|
||||
avatar_url = _first_str(room.get("owner_avatar"), room.get("avatar_mid"))
|
||||
if avatar_url:
|
||||
out["avatar_url"] = avatar_url
|
||||
if not out.get("nickname") and not out.get("avatar_url"):
|
||||
raise ValueError("douyu betard parsed empty")
|
||||
return out
|
||||
|
||||
|
||||
def fetch_douyu_profile(
|
||||
opener: urllib.request.OpenerDirector, row: dict
|
||||
) -> dict:
|
||||
"""Enrich a Douyu streamer from author URL, room→up_id, else betard."""
|
||||
profile_url = str(row.get("profile_url") or "").strip()
|
||||
live_url = str(row.get("live_url") or "").strip()
|
||||
if DOUYU_AUTHOR_HASH_RE.search(profile_url):
|
||||
return fetch_douyu_author_profile(opener, profile_url)
|
||||
room_id = None
|
||||
for candidate in (profile_url, live_url):
|
||||
m = DOUYU_ROOM_RE.search(candidate)
|
||||
if m:
|
||||
room_id = m.group(1)
|
||||
break
|
||||
if not room_id:
|
||||
raise ValueError("douyu row needs author profile_url or room live_url")
|
||||
try:
|
||||
author_url = resolve_douyu_author_url_from_room(opener, room_id)
|
||||
return fetch_douyu_author_profile(opener, author_url)
|
||||
except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
|
||||
print(f" douyu room→author failed ({e}); betard fallback", flush=True)
|
||||
return fetch_douyu_room_profile(room_id)
|
||||
|
||||
|
||||
def fetch_douyin_profile(
|
||||
opener: urllib.request.OpenerDirector, profile_url: str
|
||||
) -> dict:
|
||||
@@ -382,6 +561,15 @@ def merge_profile(row: dict, profile: dict, *, streamer_id: str) -> None:
|
||||
if val is None or val == "":
|
||||
continue
|
||||
row[key] = val
|
||||
# Fill missing live/profile URLs from platform enrichment; never wipe seeds.
|
||||
for key in ("live_url", "profile_url"):
|
||||
val = profile.get(key)
|
||||
if isinstance(val, str) and val and not str(row.get(key) or "").strip():
|
||||
row[key] = val
|
||||
# Prefer canonical Douyu author homepage when enrichment found one.
|
||||
prof = profile.get("profile_url")
|
||||
if isinstance(prof, str) and DOUYU_AUTHOR_HASH_RE.search(prof):
|
||||
row["profile_url"] = prof
|
||||
avatar_url = profile.get("avatar_url")
|
||||
if isinstance(avatar_url, str) and avatar_url:
|
||||
dest = AVATAR_DIR / f"{streamer_id}.jpg"
|
||||
@@ -408,18 +596,25 @@ def enrich_streamers(
|
||||
if ids is not None and sid not in ids:
|
||||
continue
|
||||
platform = str(row.get("platform") or "").strip().lower()
|
||||
if platform != "douyin":
|
||||
print(f"skip {sid}: platform={platform!r} (only douyin supported)", flush=True)
|
||||
if platform not in ("douyin", "douyu"):
|
||||
print(
|
||||
f"skip {sid}: platform={platform!r} "
|
||||
f"(supported: douyin, douyu)",
|
||||
flush=True,
|
||||
)
|
||||
skip += 1
|
||||
continue
|
||||
print(f"fetching {sid} ({platform}) ...", flush=True)
|
||||
try:
|
||||
if platform == "douyin":
|
||||
profile_url = str(row.get("profile_url") or "").strip()
|
||||
if not profile_url:
|
||||
print(f"skip {sid}: missing profile_url", flush=True)
|
||||
skip += 1
|
||||
continue
|
||||
print(f"fetching {sid} ...", flush=True)
|
||||
try:
|
||||
profile = fetch_douyin_profile(opener, profile_url)
|
||||
else:
|
||||
profile = fetch_douyu_profile(opener, row)
|
||||
merge_profile(row, profile, streamer_id=sid)
|
||||
nick = row.get("nickname") or "?"
|
||||
print(
|
||||
@@ -434,7 +629,7 @@ def enrich_streamers(
|
||||
time.sleep(0.8)
|
||||
if ok > 0:
|
||||
payload["fetched_at"] = _now_iso()
|
||||
payload["source"] = payload.get("source") or "manual+douyin"
|
||||
payload["source"] = payload.get("source") or "manual+douyin+douyu"
|
||||
meta = payload.get("platform_meta")
|
||||
if not isinstance(meta, dict):
|
||||
meta = {}
|
||||
@@ -442,13 +637,19 @@ def enrich_streamers(
|
||||
"douyin",
|
||||
{"label_zh": "抖音", "icon": "ui-icon/platform_douyin.png"},
|
||||
)
|
||||
meta.setdefault(
|
||||
"douyu",
|
||||
{"label_zh": "斗鱼", "icon": "ui-icon/platform_douyu.png"},
|
||||
)
|
||||
payload["platform_meta"] = meta
|
||||
_ = PROFILE_KEYS
|
||||
return ok, skip, fail
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="Enrich streamers.json from Douyin profiles")
|
||||
ap = argparse.ArgumentParser(
|
||||
description="Enrich streamers.json from Douyin / Douyu profiles"
|
||||
)
|
||||
ap.add_argument("--out", type=Path, default=OUT)
|
||||
ap.add_argument(
|
||||
"--ids",
|
||||
|
||||
+15
-1
@@ -4364,7 +4364,14 @@ function buildStreamerMetaLine(row) {
|
||||
const parts = [];
|
||||
if (uniqueId) {
|
||||
parts.push({
|
||||
label: platform === "douyin" ? "抖音号" : platform === "bilibili" ? "UID" : "账号",
|
||||
label:
|
||||
platform === "douyin"
|
||||
? "抖音号"
|
||||
: platform === "bilibili"
|
||||
? "UID"
|
||||
: platform === "douyu"
|
||||
? "房间号"
|
||||
: "账号",
|
||||
value: String(uniqueId),
|
||||
account: true,
|
||||
});
|
||||
@@ -4373,6 +4380,13 @@ function buildStreamerMetaLine(row) {
|
||||
if (Number.isFinite(Number(row.following_count))) {
|
||||
parts.push({ label: "关注", value: formatStreamerCount(row.following_count) });
|
||||
}
|
||||
} else if (platform === "douyu") {
|
||||
if (Number.isFinite(Number(row.following_count))) {
|
||||
parts.push({ label: "关注", value: formatStreamerCount(row.following_count) });
|
||||
}
|
||||
if (Number.isFinite(Number(row.total_favorited))) {
|
||||
parts.push({ label: "播放", value: formatStreamerCount(row.total_favorited) });
|
||||
}
|
||||
} else if (Number.isFinite(Number(row.total_favorited))) {
|
||||
parts.push({ label: "获赞", value: formatStreamerCount(row.total_favorited) });
|
||||
}
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/* Local defaults; production export overwrites via export_relations_site.py. */
|
||||
var SITE_VERSION = "0.5.112";
|
||||
var SITE_VERSION = "0.5.114";
|
||||
var ABILITY_VIDEO_BASE = "";
|
||||
var STATIC_ASSET_BASE = "";
|
||||
|
||||
|
||||
@@ -10,6 +10,8 @@
|
||||
* Probe logic is a JS port of fetch_streamer_live.py:
|
||||
* - Bilibili: api.live.bilibili.com Room/get_info; data.live_status === 1 is
|
||||
* live (0 offline, 2 replay counts as offline).
|
||||
* - Douyu: www.douyu.com/betard/<room_id>; room.show_status === 1 is live
|
||||
* (2 offline); room.videoLoop === 1 (carousel) counts as offline.
|
||||
* - Douyin: warm up cookies (www.douyin.com + live.douyin.com), then GET
|
||||
* live.douyin.com/<web_rid> with a browser UA and parse the escaped JSON in
|
||||
* the SSR pace chunks: roomStore.roomInfo.room.status (2 live / 4 offline);
|
||||
@@ -44,6 +46,7 @@ const DOUYIN_HOME = "https://www.douyin.com/";
|
||||
const DOUYIN_LIVE_HOME = "https://live.douyin.com/";
|
||||
const BILIBILI_INFO_URL =
|
||||
"https://api.live.bilibili.com/room/v1/Room/get_info?room_id=";
|
||||
const DOUYU_BETARD_URL = "https://www.douyu.com/betard/";
|
||||
|
||||
// Escaped JSON inside the SSR pace chunks: \"roomStore\":{\"roomInfo\":{\"room\":{
|
||||
const DOUYIN_ROOMSTORE_RE = /\\"roomStore\\":\s*\{\\"roomInfo\\":\s*\{\\"room\\":\s*\{/;
|
||||
@@ -162,7 +165,25 @@ export async function probeBilibiliRoom(roomId) {
|
||||
return data.live_status === 1;
|
||||
}
|
||||
|
||||
/** First path segment of the live room URL (douyin web_rid / bilibili room id). */
|
||||
/** show_status: 1 live, 2 offline; videoLoop carousel counts as offline. */
|
||||
export async function probeDouyuRoom(roomId) {
|
||||
const res = await fetchWithTimeout(DOUYU_BETARD_URL + roomId, {
|
||||
headers: {
|
||||
"User-Agent": BROWSER_UA,
|
||||
Accept: "application/json",
|
||||
Referer: "https://www.douyu.com/",
|
||||
},
|
||||
});
|
||||
const payload = await res.json();
|
||||
const room = payload && payload.room;
|
||||
if (!room || typeof room !== "object") {
|
||||
throw new Error("douyu betard returned no room");
|
||||
}
|
||||
if (Number(room.videoLoop || 0) === 1) return false;
|
||||
return Number(room.show_status) === 1;
|
||||
}
|
||||
|
||||
/** First path segment of the live room URL (douyin/bilibili/douyu room id). */
|
||||
export function roomRefFromUrl(liveUrl) {
|
||||
let path = "";
|
||||
try {
|
||||
@@ -180,6 +201,7 @@ export function livePlatformFromUrl(liveUrl, fallback = "") {
|
||||
const host = new URL(String(liveUrl).trim()).hostname.toLowerCase();
|
||||
if (host.includes("bilibili.com")) return "bilibili";
|
||||
if (host.includes("douyin.com")) return "douyin";
|
||||
if (host.includes("douyu.com")) return "douyu";
|
||||
} catch {
|
||||
/* keep fallback */
|
||||
}
|
||||
@@ -245,6 +267,7 @@ async function probeAll(targets) {
|
||||
let isLive;
|
||||
if (t.platform === "douyin") isLive = await probeDouyinRoom(t.ref, jar);
|
||||
else if (t.platform === "bilibili") isLive = await probeBilibiliRoom(t.ref);
|
||||
else if (t.platform === "douyu") isLive = await probeDouyuRoom(t.ref);
|
||||
else throw new Error(`unsupported platform ${t.platform}`);
|
||||
results.set(t.id, { is_live: isLive });
|
||||
} catch (err) {
|
||||
|
||||
@@ -5,8 +5,8 @@
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1" />
|
||||
<title>DOTA2 上分帝</title>
|
||||
<link rel="icon" href="/ui-icon/dota2_logo.png" type="image/png" />
|
||||
<link rel="stylesheet" href="/style.css?v=0.5.112" />
|
||||
<script src="/mobile-gate.js?v=0.5.112"></script>
|
||||
<link rel="stylesheet" href="/style.css?v=0.5.114" />
|
||||
<script src="/mobile-gate.js?v=0.5.114"></script>
|
||||
</head>
|
||||
<body>
|
||||
<h1 class="sr-only">DOTA2 上分帝</h1>
|
||||
@@ -178,8 +178,8 @@
|
||||
|
||||
<section class="detail" id="detail" aria-live="polite"></section>
|
||||
|
||||
<script src="/config.js?v=0.5.112"></script>
|
||||
<script src="/router.js?v=0.5.112"></script>
|
||||
<script src="/app.js?v=0.5.112"></script>
|
||||
<script src="/config.js?v=0.5.114"></script>
|
||||
<script src="/router.js?v=0.5.114"></script>
|
||||
<script src="/app.js?v=0.5.114"></script>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
@@ -17,6 +17,12 @@ sys.path.insert(0, str(ROOT / "web"))
|
||||
from web import notify_site_traffic, refresh_web
|
||||
from web.fetch_stratz_meta import preserve_failed_meta_brackets
|
||||
from web.fetch_streamer_live import probe_streamers
|
||||
from web.fetch_streamers import (
|
||||
DOUYU_AUTHOR_HASH_RE,
|
||||
extract_douyu_up_id_from_room_html,
|
||||
extract_profile_from_douyu_data,
|
||||
parse_douyu_dollar_data,
|
||||
)
|
||||
from web.refresh_cache import restore_patches, restore_streamers
|
||||
from shared.http_utils import write_json_atomic
|
||||
from shared.paths import HEROES_JSON, RELATIONS_JSON, WEB_FRONTEND
|
||||
@@ -260,6 +266,65 @@ class LiveProbeFallbackTests(unittest.TestCase):
|
||||
self.assertFalse(row["is_live"])
|
||||
self.assertNotIn("live_probed_at", row)
|
||||
|
||||
def test_douyu_probe_path_writes_live_flag(self) -> None:
|
||||
payload = {
|
||||
"streamers": [
|
||||
{
|
||||
"id": "gouhuang",
|
||||
"platform": "douyu",
|
||||
"live_url": "https://www.douyu.com/957090",
|
||||
}
|
||||
]
|
||||
}
|
||||
with patch("web.fetch_streamer_live.probe_douyu", return_value=True):
|
||||
live, offline, fail = probe_streamers(payload)
|
||||
self.assertEqual((live, offline, fail), (1, 0, 0))
|
||||
self.assertTrue(payload["streamers"][0]["is_live"])
|
||||
self.assertIn("live_probed_at", payload["streamers"][0])
|
||||
|
||||
|
||||
class DouyuProfileParseTests(unittest.TestCase):
|
||||
def test_author_and_author_video_hashes(self) -> None:
|
||||
self.assertEqual(
|
||||
DOUYU_AUTHOR_HASH_RE.search(
|
||||
"https://v.douyu.com/author/JPw9YOLKlw5X"
|
||||
).group(1),
|
||||
"JPw9YOLKlw5X",
|
||||
)
|
||||
self.assertEqual(
|
||||
DOUYU_AUTHOR_HASH_RE.search(
|
||||
"https://v.douyu.com/author-video/JPw9YOLKlw5X"
|
||||
).group(1),
|
||||
"JPw9YOLKlw5X",
|
||||
)
|
||||
|
||||
def test_dollar_data_maps_fan_fields(self) -> None:
|
||||
blob = (
|
||||
'{type:"0",uid:"13506421",upId:"JPw9YOLKlw5X",upFollowNum:"16",'
|
||||
'subscribeNum:"465781",playCount:"26074077",'
|
||||
'avatar:"https://example.com/a.jpg",'
|
||||
'name: "18yearsold\\u5929\\u6b8b\\u5c11\\u5e74K9",'
|
||||
'contents: "",roomId:"235520",roomName: "k9 room"}'
|
||||
)
|
||||
profile = extract_profile_from_douyu_data(parse_douyu_dollar_data(blob))
|
||||
self.assertEqual(profile["nickname"], "18yearsold天残少年K9")
|
||||
self.assertEqual(profile["unique_id"], "235520")
|
||||
self.assertEqual(profile["following_count"], 16)
|
||||
self.assertEqual(profile["follower_count"], 465781)
|
||||
self.assertEqual(profile["total_favorited"], 26074077)
|
||||
self.assertEqual(profile["profile_url"], "https://v.douyu.com/author/JPw9YOLKlw5X")
|
||||
self.assertEqual(profile["live_url"], "https://www.douyu.com/235520")
|
||||
self.assertNotIn("signature", profile)
|
||||
|
||||
def test_room_html_up_id_extraction(self) -> None:
|
||||
plain = '{"rid":957090,"up_id":"EqAvg1lQD75L","ban_display":0}'
|
||||
escaped = (
|
||||
r'{\"rid\":957090,\"up_id\":\"EqAvg1lQD75L\",\"ban_display\":0}'
|
||||
)
|
||||
self.assertEqual(extract_douyu_up_id_from_room_html(plain), "EqAvg1lQD75L")
|
||||
self.assertEqual(extract_douyu_up_id_from_room_html(escaped), "EqAvg1lQD75L")
|
||||
self.assertIsNone(extract_douyu_up_id_from_room_html("<html></html>"))
|
||||
|
||||
|
||||
class NotifySummaryTests(unittest.TestCase):
|
||||
def test_skipped_summary_is_labeled(self) -> None:
|
||||
|
||||
Reference in New Issue
Block a user