diff --git a/AGENTS.md b/AGENTS.md index a73ca66..9c02ed2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -88,9 +88,9 @@ climperor/ | `web/fetch_hero_matches.py` | 拉取同英雄近期比赛 + 终局出装/加点 → `web/data/hero_matches.json`（`--source league\|public\|both`；合并后保留最近 N 场胜局，默认 10；天梯需传奇及以上；公开列表过滤 bot/Turbo、仅 ranked lobby；`--public-region china` 优先国服；`--workers` 详情并发 + 凑满即停；可选 `OPENDOTA_API_KEY`；增量补缺失/不足 N 场；`--enrich-item-times` 补购买时间；**仅上分帝 Web**；勿进 recommend） | | `web/fetch_pro_matches.py` | 按明星名单拉近期联赛/锦标赛对局 → `web/data/pro_matches.json`（默认读 `web/data/pro_player_watchlist.json`；`--include-pubs` 另拉天梯 lobby 7；每种 lobby 各保留 `--limit` 场；`--players` 覆盖整文件；`--all-pros` 从 `/proPlayers` 盲抽；含终局出装/加点；进 `refresh_web` daily 且带 `--include-pubs`；**仅上分帝 Web「比赛」**；勿进 recommend） | | `web/fetch_leaderboards.py` | 拉取 Valve Immortal 四区榜 Top100 → `web/data/leaderboards.json`（**仅上分帝 Web「排行」**；无 MMR/account_id；勿进 recommend） | -| `web/fetch_streamers.py` | 从抖音主页补全 `web/data/streamers.json` 的昵称/签名/关注粉丝获赞/头像（手工名单 + 直播间/主页 URL；支持 `v.douyin.com` 短链；失败保留旧值；**不**探测开播（由 `fetch_streamer_live.py` 负责）；**仅上分帝 Web「主播」**；勿进 recommend；进 `refresh_web` daily） | -| `web/fetch_streamer_live.py` | 探测主播真实在播状态回写 `web/data/streamers.json` 的 `is_live`/`live_probed_at`：抖音解析直播间 SSR 页 `roomStore.roomInfo.room.status`（2 在播 / 4 下播；预热 cookie + ~1s 间隔；web_rid 校验），B 站走 `Room/get_info`（`live_status==1` 在播，轮播算下播）；探测失败清为 `is_live:false` 并去掉 `live_probed_at`（与 `/api/live-status` 一致，不沿用旧直播中）、始终 exit 0；`--ids a,b` 限范围、`--dry-run` 只打印；仅 Web；进 `refresh_web` daily（角标以访问触发的 live API 为准，daily 仅作 data.json 兜底） | -| `web/frontend/functions/api/live-status.js` | Pages Function `GET /api/live-status`：访问触发的在播探测（逻辑同 `fetch_streamer_live.py`），读 `data.json` 的 `streamers.streamers`；Cache API 固定键 + isolate 内 in-flight 合并（5 分钟新鲜窗口，**无 KV**）；抖音从数据中心 IP 失败属预期 → 失败主播一律 `is_live:false` + `stale:true`（**不**沿用旧的直播中）；全失败回 `stale-override` 空角标表或 `error`，永不 500；导出时拷贝 `functions/`；**部署须 `cwd=dist` 跑 wrangler**（Functions 相对 cwd 解析） | +| `web/fetch_streamers.py` | 从抖音 / 斗鱼主页补全 `web/data/streamers.json` 的昵称/签名/关注粉丝获赞(播放)/头像（手工名单 + 直播间/主页 URL；抖音支持 `v.douyin.com` 短链；斗鱼优先 `v.douyu.com/author/`（兼容 `author-video`）的 `$DATA`，仅房间号时从直播间 HTML 解析 `up_id` 再拉作者页，失败才回退 `betard`；失败保留旧值；**不**探测开播（由 `fetch_streamer_live.py` 负责）；**仅上分帝 Web「主播」**；勿进 recommend；进 `refresh_web` daily） | +| `web/fetch_streamer_live.py` | 探测主播真实在播状态回写 `web/data/streamers.json` 的 `is_live`/`live_probed_at`：抖音解析直播间 SSR 页 `roomStore.roomInfo.room.status`（2 在播 / 4 下播；预热 cookie + ~1s 间隔；web_rid 校验），B 站走 `Room/get_info`（`live_status==1` 在播，轮播算下播），斗鱼走 `betard/{room_id}`（`show_status==1` 在播，`videoLoop==1` 轮播算下播）；探测失败清为 `is_live:false` 并去掉 `live_probed_at`（与 `/api/live-status` 一致，不沿用旧直播中）、始终 exit 0；`--ids a,b` 限范围、`--dry-run` 只打印；仅 Web；进 `refresh_web` daily（角标以访问触发的 live API 为准，daily 仅作 data.json 兜底） | +| `web/frontend/functions/api/live-status.js` | Pages Function `GET /api/live-status`：访问触发的在播探测（逻辑同 `fetch_streamer_live.py`，含抖音 / B 站 / 斗鱼），读 `data.json` 的 `streamers.streamers`；Cache API 固定键 + isolate 内 in-flight 合并（5 分钟新鲜窗口，**无 KV**）；抖音从数据中心 IP 失败属预期 → 失败主播一律 `is_live:false` + `stale:true`（**不**沿用旧的直播中）；全失败回 `stale-override` 空角标表或 `error`，永不 500；导出时拷贝 `functions/`；**部署须 `cwd=dist` 跑 wrangler**（Functions 相对 cwd 解析） | | `web/frontend/functions/api/mobile-demand.js` | Pages Function `GET\|POST /api/mobile-demand`：移动端「催更」需求计数（Cache API 存 `count`，**无 KV**；边缘竞态/驱逐可能少计或重置）；本机 `serve_relations.py` 写 `web/.refresh/mobile_demand.json`；前端 `mobile-gate.js` 用 UA 识别手机/平板并拦截，`localStorage` 同设备只 POST 一次 | | `web/frontend/mobile-gate.js` | 移动端门禁（`` 早载）：`html.mobile-client` + 催更按钮；设 `window.__CLIMPEROR_MOBILE__` 供 `app.js` 跳过桌面 boot | | `web/fetch_item_shop.py` | 官网商店 11 列目录（dota2.com.cn/itemscategory）+ 合成图 → `web/data/item_shop.json` + 图标 | @@ -109,7 +109,7 @@ climperor/ | `web/data/pro_player_watchlist.json` | 明星选手 OpenDota `account_id` 手工名单（对照 Dotabuff / Liquipedia；含现役战队席位与昔日国服明星；外号：查理斯→Chalice，Somnus 即 Maybe；CN/EU/SA 等）；`fetch_pro_matches.py` 默认只拉此名单；队名可过期，以 id 为准；**不**进 recommend | | `web/data/pro_matches.json` | 明星选手近期联赛/锦标赛（可选天梯）对局（`by_pro`/`by_hero` + 终局出装/加点 + `lobby_type`/`origin`；由 `fetch_pro_matches.py` 生成；Web 顶层「比赛」侧栏筛「全部 / 职业 / 国服」+ 选手，与英雄详情「近期比赛」合并展示；进 daily；**不**参与局内推荐） | | `web/data/leaderboards.json` | Valve Immortal 四区 Top100（`china`/`europe`/`americas`/`se_asia`；仅排名+昵称等；由 `fetch_leaderboards.py` 生成；Web「排行」选手榜只读；**不**参与局内推荐） | -| `web/data/streamers.json` | 主播目录（手工 `live_url`/profile URL + 常用英雄 + 可选精选视频 `video`/`video_title` + 可选 `video_poster`/`video_fit`/`video_crop`/`video_aspect`；抖音/B 站 profile 补全；`platform_meta`；Web 卡片：头像行 = 头像 \| 昵称+账号/获赞粉丝 \|「关注」，签名（`signature`）独立全宽行（最长 3 行）；在播时粉环 +「直播」角标叠在环底（抖音式 `bottom:-6px`，无间距）；列表排序：先 `is_live` 再粉丝数降序；视口分档滚播（远处不拉、近处 metadata、中部 `canplay` 且单路 `preload=auto`）+ 同名 JPG 封面占位；有 `live_url` 时点头像进直播间；**「直播」角标只信本轮成功探测**（线上 `/api/live-status` 边缘缓存 5 分钟；本地 `serve_relations` 同逻辑缓存 60s；`stale`/失败不显示角标；`data.json` daily 仅作首屏兜底直至接口返回；`live_url` 仅作点击入口）；由 `fetch_streamers.py` 补全；Web「主播」只读；**不**参与局内推荐） | +| `web/data/streamers.json` | 主播目录（手工 `live_url`/profile URL + 常用英雄 + 可选精选视频 `video`/`video_title` + 可选 `video_poster`/`video_fit`/`video_crop`/`video_aspect`；抖音 profile 补全；平台含抖音 / B 站 / 斗鱼；`platform_meta`；Web 卡片：头像行 = 头像 \| 昵称+账号/获赞粉丝 \|「关注」，签名（`signature`）独立全宽行（最长 3 行）；在播时粉环 +「直播」角标叠在环底（抖音式 `bottom:-6px`，无间距）；列表排序：先 `is_live` 再粉丝数降序；视口分档滚播（远处不拉、近处 metadata、中部 `canplay` 且单路 `preload=auto`）+ 同名 JPG 封面占位；有 `live_url` 时点头像进直播间；**「直播」角标只信本轮成功探测**（线上 `/api/live-status` 边缘缓存 5 分钟；本地 `serve_relations` 同逻辑缓存 60s；`stale`/失败不显示角标；`data.json` daily 仅作首屏兜底直至接口返回；`live_url` 仅作点击入口）；由 `fetch_streamers.py` 补全；Web「主播」只读；**不**参与局内推荐） | | `web/data/stratz_hero_meta.json` | STRATZ 各勋章段位近 N 周 pick/win（`weeks`/`latest`）+ **同段位最近 1 周分路**（`winWeek`+`positionIds`）+ `meta_board`（由 `fetch_stratz_meta.py` 生成；Web 英雄详情「走势」优先 + 顶层「走势」`#/trends` 近 N 周榜；**不**参与局内推荐） | | `web/data/stratz_matchup_tops.json` | STRATZ 对位/协同 Top（counters/countered/synergies；由 `fetch_stratz_meta.py` 生成；全局聚合 + `scope`/`stale`/`fetched_at`；Web「对位」Tab；与定性 `relations.json` 分开展示；**不**参与局内推荐） | | `web/data/item_shop.json` | 商店 11 列目录（官网 basic/upgrade；由 `fetch_item_shop.py` 生成；物品页只读） | @@ -163,7 +163,7 @@ Dota 2 GSI → pc/gsi_watch.py (:3223) - **合规**：仅 GSI + 屏幕截图；**禁止**读进程内存、注入、绕过 VAC。 - **GSI 范围**：普通玩家视角拿不到双方 pick；GSI 只作阶段触发、`team_slot`、本人 `hero`、本机 `accountid`/`steamid`。本人顶栏槽位**只信** GSI，不用截屏名字亮度猜测。 -- **克制 / 搭档数据**：机制克制/搭档只用定性边存 `shared/data/relations.json`（克制有向 + 搭档无向 + 理由），**不要**用胜率/场次表达机制克制，也**不要**写入 `shared/data/heroes.json`。Web 英雄页三视图：克制 / 被克制 / 搭档。OpenDota 段位胜率/场次单独存 `web/data/hero_stats.json`（近约 7 天窗；上场率 = 出场/(Σ出场/10)；冠绝样本过小时与超凡合并），**仅**上分帝 Web 英雄详情「走势」**无 STRATZ 时兜底**。STRATZ 周胜率/分路/`meta_board` 存 `web/data/stratz_hero_meta.json`（三卡与分路均取 **当前选中勋章 · 最近 1 周** `winWeek`；近 8 周列表从新到旧；分路在 8 周列表上方），对位 Top 存 `web/data/stratz_matchup_tops.json`（英雄详情「走势」与顶层「走势」榜 `#/trends` 优先用 STRATZ；「对位」Tab 只读），**禁止**合并进 relations/heroes，禁止 `pc/recommend.py` 读取。同英雄近期比赛出装/加点单独存 `web/data/hero_matches.json`，**仅** Web「近期比赛」Tab，禁止进 recommend。Valve Immortal 四区榜单独存 `web/data/leaderboards.json`，**仅** Web「排行」页选手榜，禁止进 recommend。主播目录单独存 `web/data/streamers.json`（手工名单 + 抖音 profile 补全；首版仅抖音），**仅** Web 顶层「主播」与英雄详情「主播」Tab，禁止进 recommend；由 `refresh_web` daily 软失败刷新粉丝等字段（不阻断整档）。 +- **克制 / 搭档数据**：机制克制/搭档只用定性边存 `shared/data/relations.json`（克制有向 + 搭档无向 + 理由），**不要**用胜率/场次表达机制克制，也**不要**写入 `shared/data/heroes.json`。Web 英雄页三视图：克制 / 被克制 / 搭档。OpenDota 段位胜率/场次单独存 `web/data/hero_stats.json`（近约 7 天窗；上场率 = 出场/(Σ出场/10)；冠绝样本过小时与超凡合并），**仅**上分帝 Web 英雄详情「走势」**无 STRATZ 时兜底**。STRATZ 周胜率/分路/`meta_board` 存 `web/data/stratz_hero_meta.json`（三卡与分路均取 **当前选中勋章 · 最近 1 周** `winWeek`；近 8 周列表从新到旧；分路在 8 周列表上方），对位 Top 存 `web/data/stratz_matchup_tops.json`（英雄详情「走势」与顶层「走势」榜 `#/trends` 优先用 STRATZ；「对位」Tab 只读），**禁止**合并进 relations/heroes，禁止 `pc/recommend.py` 读取。同英雄近期比赛出装/加点单独存 `web/data/hero_matches.json`，**仅** Web「近期比赛」Tab，禁止进 recommend。Valve Immortal 四区榜单独存 `web/data/leaderboards.json`，**仅** Web「排行」页选手榜，禁止进 recommend。主播目录单独存 `web/data/streamers.json`（手工名单 + 抖音 profile 补全；平台含抖音 / B 站 / 斗鱼），**仅** Web 顶层「主播」与英雄详情「主播」Tab，禁止进 recommend；由 `refresh_web` daily 软失败刷新粉丝等字段（不阻断整档）。 - **装备机制 / 怕的装备**：标签与技能汇总来自 Valve/OpenDota 自动抽取 + `item_tag_overrides.json`；`item_fears.py` 规则映射到英雄弱点。`fetch_item_counter_stats.py` 的对阵购买率/胜率差仅作为观测证据：须减去同装备全局基线，对全部英雄的已有机制候选只允许小幅调序；统计发现的新组合即使同时满足 `games≥100`、购买率提升 `≥3pp`、条件胜率差 `≥1.5pp`、两项双比例检验 `z≥1.96`，仍须确认机制成立后手工写入 overrides，禁止直接把高相关当因果克制。大段技能文案只进 `web/data/hero_abilities.json` / `items_meta.json`，**不要**塞进 `heroes.json`。本阶段仅上分帝 Web 展示，**不对局内出装推荐**。核心装「使用率」为 `hero_items` 列表内相对热度归一化，非绝对出场率、无段位维度。技能/物品「施加」类 tags（机制查询页）与技能 `dispellable`（效果能否被驱散）严格区分；**禁止**写入 recommend / relations。 - **机制查询页**：顶层 `#/mechanics[/{effect}]`；侧栏弱/强驱散 + 核心控制；结果为施加该效果的技能与物品。数据边界：仅 Web。 - **模板策略**：只维护 CDN 层。皮肤问题用会话策略（选人可改判、决策 `allow_revise=False`、best 帧偏 HERO_SELECTION），不要为皮肤加模板库，也不要复活 real 双层库。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 19749e1..4f2946a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,13 @@ ## [Unreleased] +## [0.5.114] - 2026-07-30 + +### Added + +- 主播资料补全支持斗鱼：`v.douyu.com/author/`（兼容 `author-video`）拉粉丝/关注/播放/头像；仅有房间号时从直播间 HTML 解析 `up_id` 再拉作者页（失败才回退 `betard`）。斗鱼卡片展示「关注 / 播放 / 粉丝」。 +- 主播页收录斗鱼「踏上征途167（狗皇）」「天残少年K9」「叶子长青K」、抖音「浙大小鱼王—dota2」与 B 站「Dota2绝中绝」；开播探测支持斗鱼（`betard`）。 + ## [0.5.112] - 2026-07-30 ### Changed diff --git a/web/assets/streamer_avatars/gouhuang.jpg b/web/assets/streamer_avatars/gouhuang.jpg new file mode 100644 index 0000000..25dfbb7 Binary files /dev/null and b/web/assets/streamer_avatars/gouhuang.jpg differ diff --git a/web/assets/streamer_avatars/juezhongjue.jpg b/web/assets/streamer_avatars/juezhongjue.jpg new file mode 100644 index 0000000..96fa648 Binary files /dev/null and b/web/assets/streamer_avatars/juezhongjue.jpg differ diff --git a/web/assets/streamer_avatars/k9.jpg b/web/assets/streamer_avatars/k9.jpg new file mode 100644 index 0000000..80fc5e7 Binary files /dev/null and b/web/assets/streamer_avatars/k9.jpg differ diff --git a/web/assets/streamer_avatars/yezi.jpg b/web/assets/streamer_avatars/yezi.jpg new file mode 100644 index 0000000..9568a6d Binary files /dev/null and b/web/assets/streamer_avatars/yezi.jpg differ diff --git a/web/assets/streamer_avatars/zhedaxiaoyuwang.jpg b/web/assets/streamer_avatars/zhedaxiaoyuwang.jpg new file mode 100644 index 0000000..9357595 Binary files /dev/null and b/web/assets/streamer_avatars/zhedaxiaoyuwang.jpg differ diff --git a/web/assets/ui_icons/platform_douyu.png b/web/assets/ui_icons/platform_douyu.png new file mode 100644 index 0000000..1484c53 Binary files /dev/null and b/web/assets/ui_icons/platform_douyu.png differ diff --git a/web/data/streamers.json b/web/data/streamers.json index 3e6b712..7d37fba 100644 --- a/web/data/streamers.json +++ b/web/data/streamers.json @@ -1,6 +1,6 @@ { - "fetched_at": "2026-07-28T14:20:00+00:00", - "source": "manual+douyin+bilibili", + "fetched_at": "2026-07-29T18:37:21.491922+00:00", + "source": "manual+douyin+bilibili+douyu", "platform_meta": { "douyin": { "label_zh": "抖音", @@ -9,6 +9,10 @@ "bilibili": { "label_zh": "哔哩哔哩", "icon": "ui-icon/platform_bilibili.png" + }, + "douyu": { + "label_zh": "斗鱼", + "icon": "ui-icon/platform_douyu.png" } }, "streamers": [ @@ -480,6 +484,102 @@ "profile_fetched_at": "2026-07-29T09:54:00+00:00", "is_live": true, "live_probed_at": "2026-07-29T09:54:00+00:00" + }, + { + "id": "gouhuang", + "platform": "douyu", + "live_url": "https://www.douyu.com/957090", + "profile_url": "https://v.douyu.com/author/EqAvg1lQD75L", + "tagline": "米波绝活 · 狗皇", + "heroes": [ + "meepo", + "broodmother", + "alchemist" + ], + "nickname": "踏上征途167", + "unique_id": "957090", + "signature": "娱乐DOTA 轻松氛围", + "avatar": "streamer_avatars/gouhuang.jpg", + "is_live": false, + "live_probed_at": "2026-07-29T18:16:27.171378+00:00", + "profile_fetched_at": "2026-07-29T18:37:18.228078+00:00", + "following_count": 868, + "follower_count": 280201, + "total_favorited": 9673470 + }, + { + "id": "juezhongjue", + "platform": "bilibili", + "live_url": "https://live.bilibili.com/1747114599", + "profile_url": "https://space.bilibili.com/520653835", + "tagline": "绝活打野炼金", + "heroes": [ + "alchemist" + ], + "nickname": "Dota2绝中绝", + "unique_id": "520653835", + "signature": "绝活打野炼金。鄙人不擅长对线对线，但极其擅长与野怪掰头！", + "follower_count": 433, + "avatar": "streamer_avatars/juezhongjue.jpg", + "is_live": false, + "live_probed_at": "2026-07-29T18:08:20.096701+00:00" + }, + { + "id": "k9", + "platform": "douyu", + "live_url": "https://www.douyu.com/235520", + "profile_url": "https://v.douyu.com/author/JPw9YOLKlw5X", + "tagline": "绝活卡尔", + "heroes": [ + "invoker" + ], + "nickname": "18yearsold天残少年K9", + "unique_id": "235520", + "signature": "k9：午夜牢车教室！.", + "avatar": "streamer_avatars/k9.jpg", + "is_live": true, + "live_probed_at": "2026-07-29T18:16:27.995333+00:00", + "following_count": 16, + "follower_count": 465781, + "total_favorited": 26074078, + "profile_fetched_at": "2026-07-29T18:23:59.723287+00:00" + }, + { + "id": "yezi", + "platform": "douyu", + "live_url": "https://www.douyu.com/246195", + "profile_url": "https://v.douyu.com/author/Kqy70jkrpdXG", + "tagline": "绝活火猫", + "heroes": [ + "ember_spirit" + ], + "nickname": "叶子长青K", + "unique_id": "246195", + "signature": "QQ:394867894 网易云搜用户 淡然听听歌", + "avatar": "streamer_avatars/yezi.jpg", + "is_live": false, + "live_probed_at": "2026-07-29T18:16:28.416689+00:00", + "profile_fetched_at": "2026-07-29T18:37:20.691635+00:00", + "following_count": 134, + "follower_count": 286391, + "total_favorited": 6336746 + }, + { + "id": "zhedaxiaoyuwang", + "platform": "douyin", + "profile_url": "https://v.douyin.com/tSe_vsZ8r7o/", + "tagline": "小鱼人", + "heroes": [ + "slark" + ], + "nickname": "浙大小鱼王—dota2", + "unique_id": "89748079578", + "signature": "本硕985，浙大在读 9700冲万分", + "following_count": 52, + "follower_count": 1802, + "total_favorited": 842, + "avatar": "streamer_avatars/zhedaxiaoyuwang.jpg", + "profile_fetched_at": "2026-07-29T18:26:00+00:00" } ] } diff --git a/web/export_relations_site.py b/web/export_relations_site.py index 6cc4b10..1edf695 100644 --- a/web/export_relations_site.py +++ b/web/export_relations_site.py @@ -52,7 +52,7 @@ from shared.paths import ( from serve_relations import WEB_DIR, build_payload -SITE_VERSION = "0.5.112" +SITE_VERSION = "0.5.114" DEFAULT_OSS_BASE = "https://climperor.oss-cn-shanghai.aliyuncs.com" diff --git a/web/fetch_streamer_live.py b/web/fetch_streamer_live.py index bab579c..3ee2495 100644 --- a/web/fetch_streamer_live.py +++ b/web/fetch_streamer_live.py @@ -13,6 +13,9 @@ Approach (verified 2026-07): - Bilibili: public API ``api.live.bilibili.com/room/v1/Room/get_info`` with the numeric room id taken from the ``live_url`` path; ``data.live_status == 1`` means live (0 offline, 2 replay — replay is treated as offline). No login. +- Douyu: public ``www.douyu.com/betard/{room_id}``; ``room.show_status == 1`` + means live, ``== 2`` offline; ``room.videoLoop == 1`` (carousel) counts as + offline. - Douyin: one shared cookie session is warmed up (www.douyin.com + live.douyin.com), then per room we GET ``live.douyin.com/{web_rid}`` with a browser UA (the numeric ``live_url`` path segment is the ``web_rid``). The @@ -65,10 +68,12 @@ TIMEOUT = 20 # Douyin rate-limits aggressively; keep ~1s spacing between its requests. DOUYIN_SPACING = 1.0 BILIBILI_SPACING = 0.5 +DOUYU_SPACING = 0.3 DOUYIN_HOME = "https://www.douyin.com/" DOUYIN_LIVE_HOME = "https://live.douyin.com/" BILIBILI_INFO_URL = "https://api.live.bilibili.com/room/v1/Room/get_info?room_id={room_id}" +DOUYU_BETARD_URL = "https://www.douyu.com/betard/{room_id}" # Escaped JSON inside the SSR pace chunks: \"roomStore\":{\"roomInfo\":{\"room\":{ DOUYIN_ROOMSTORE_RE = re.compile( @@ -152,8 +157,27 @@ def probe_bilibili(room_id: str) -> bool: return data.get("live_status") == 1 +def probe_douyu(room_id: str) -> bool: + """show_status: 1 live, 2 offline; videoLoop carousel counts as offline.""" + payload = http_utils.http_json( + DOUYU_BETARD_URL.format(room_id=room_id), timeout=TIMEOUT + ) + if not isinstance(payload, dict): + raise ValueError("douyu betard returned non-object") + room = payload.get("room") + if not isinstance(room, dict): + raise ValueError("douyu betard returned no room") + if int(room.get("videoLoop") or 0) == 1: + return False + status = room.get("show_status") + try: + return int(status) == 1 + except (TypeError, ValueError) as e: + raise ValueError(f"douyu unexpected show_status {status!r}") from e + + def room_ref_from_url(live_url: str) -> str | None: - """First path segment of the live room URL (douyin web_rid / bilibili room id).""" + """First path segment of the live room URL (douyin/bilibili/douyu room id).""" path = urllib.parse.urlparse(live_url.strip()).path.strip("/") if not path: return None @@ -167,6 +191,8 @@ def live_platform_from_url(live_url: str, fallback: str = "") -> str: return "bilibili" if "douyin.com" in host: return "douyin" + if "douyu.com" in host: + return "douyu" return (fallback or "").strip().lower() @@ -210,6 +236,9 @@ def probe_streamers( elif platform == "bilibili": is_live = probe_bilibili(ref) time.sleep(BILIBILI_SPACING) + elif platform == "douyu": + is_live = probe_douyu(ref) + time.sleep(DOUYU_SPACING) else: print(f"skip {sid}: platform={platform!r} unsupported", flush=True) continue diff --git a/web/fetch_streamers.py b/web/fetch_streamers.py index 3a7c441..5676364 100644 --- a/web/fetch_streamers.py +++ b/web/fetch_streamers.py @@ -1,16 +1,23 @@ -"""Fetch Douyin profile fields into data/streamers.json. +"""Fetch Douyin / Douyu profile fields into data/streamers.json. Manual seed fields (id / platform / live_url / profile_url / heroes / tagline) are preserved. Profile enrichment (nickname, signature, counts, avatar) is -best-effort via Douyin HTML RENDER_DATA + text fallback; failures keep the -previous values. +best-effort: + +- Douyin: HTML RENDER_DATA + text fallback +- Douyu: ``v.douyu.com/author/`` (or ``author-video/``) page + ``window.$DATA`` (fans / following / plays / avatar). Room-only rows + resolve ``up_id`` from the live-room HTML then fetch the author page; + if that fails, fall back to ``betard`` (nickname / avatar / bio, no fans). + +Failures keep the previous values. Preview only — do not merge into relations/heroes or recommend. Part of refresh_web ``daily`` (soft-fail: never aborts the tier). Usage: python fetch_streamers.py - python fetch_streamers.py --ids xiaowang + python fetch_streamers.py --ids xiaowang k9 python fetch_streamers.py --out data/streamers.json """ @@ -32,8 +39,8 @@ import urllib.request from datetime import datetime, timezone from typing import Any -from shared.http_utils import write_json_atomic -from shared.paths import DATA, STREAMER_AVATARS +from shared.http_utils import http_json, write_json_atomic +from shared.paths import DATA, ROOT, STREAMER_AVATARS OUT = DATA / "streamers.json" AVATAR_DIR = STREAMER_AVATARS @@ -58,6 +65,17 @@ HTML_AVATAR_RE = re.compile( re.IGNORECASE, ) SEC_UID_RE = re.compile(r"/user/(MS4wLjABAAAA[A-Za-z0-9_-]+)") +DOUYU_AUTHOR_HASH_RE = re.compile( + r"(?:v\.)?douyu\.com/author(?:-video)?/([A-Za-z0-9]+)", re.IGNORECASE +) +DOUYU_ROOM_RE = re.compile( + r"(?:www\.)?douyu\.com/(\d+)(?:/|$|\?)", re.IGNORECASE +) +DOUYU_DATA_RE = re.compile(r"window\.\$DATA=(\{.*?\}),\$", re.DOTALL) +DOUYU_BARE_KEY_RE = re.compile(r"([{\s,])([A-Za-z_][A-Za-z0-9_]*)\s*:") +# Room HTML embeds up_id in plain JSON and/or JSON-escaped script strings. +DOUYU_UP_ID_RE = re.compile(r'\\?"up_id\\?"\s*:\s*\\?"([A-Za-z0-9]+)\\?"') +DOUYU_BETARD_URL = "https://www.douyu.com/betard/{room_id}" # Fields fetch may overwrite; manual seed keys are never removed. PROFILE_KEYS = ( "nickname", @@ -104,7 +122,11 @@ def _opener() -> urllib.request.OpenerDirector: def _get( - opener: urllib.request.OpenerDirector, url: str, *, timeout: int = 30 + opener: urllib.request.OpenerDirector, + url: str, + *, + timeout: int = 30, + referer: str = "https://www.douyin.com/", ) -> tuple[str, str]: """Return (final_url, html).""" req = urllib.request.Request( @@ -113,7 +135,7 @@ def _get( "User-Agent": BROWSER_UA, "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8", - "Referer": "https://www.douyin.com/", + "Referer": referer, }, ) with opener.open(req, timeout=timeout) as resp: @@ -306,7 +328,7 @@ def resolve_profile_url( return url, None -def download_avatar(url: str, dest: Path) -> bool: +def download_avatar(url: str, dest: Path, *, referer: str | None = None) -> bool: dest.parent.mkdir(parents=True, exist_ok=True) # Prefer a larger CDN variant when the URL embeds a size token. candidates = [url] @@ -314,12 +336,22 @@ def download_avatar(url: str, dest: Path) -> bool: candidates.insert(0, url.replace("/100x100/", "/720x720/")) if "300x300" in url: candidates.insert(0, url.replace("300x300", "720x720")) + if "_avatar_middle." in url: + candidates.insert(0, url.replace("_avatar_middle.", "_avatar_big.")) + if "_middle.jpg" in url: + candidates.insert(0, url.replace("_middle.jpg", "_big.jpg")) + host = urllib.parse.urlparse(url).netloc.lower() + if referer is None: + if "douyu" in host: + referer = "https://www.douyu.com/" + else: + referer = "https://www.douyin.com/" for candidate in candidates: req = urllib.request.Request( candidate, headers={ "User-Agent": BROWSER_UA, - "Referer": "https://www.douyin.com/", + "Referer": referer, }, ) try: @@ -336,6 +368,153 @@ def download_avatar(url: str, dest: Path) -> bool: return False +def parse_douyu_dollar_data(blob: str) -> dict: + """Parse Douyu ``window.$DATA={...}`` JS object (bare keys) into a dict.""" + quoted = DOUYU_BARE_KEY_RE.sub(r'\1"\2":', blob.strip()) + data = json.loads(quoted) + if not isinstance(data, dict): + raise ValueError("douyu $DATA is not an object") + return data + + +def extract_profile_from_douyu_data(data: dict) -> dict: + """Map Douyu author ``$DATA`` fields onto streamer profile keys.""" + out: dict[str, Any] = {} + nick = _first_str(data.get("name"), data.get("nickname")) + if nick: + out["nickname"] = nick + room_id = _first_str(data.get("roomId"), data.get("room_id")) + if room_id: + out["unique_id"] = room_id + out["live_url"] = f"https://www.douyu.com/{room_id}" + up_id = _first_str(data.get("upId"), data.get("up_id")) + if up_id: + # Canonical homepage (author space); author-video is an alias. + out["profile_url"] = f"https://v.douyu.com/author/{up_id}" + # Author bio only — do not fall back to live room title (would wipe + # hand-seeded / betard signatures on every refresh). + contents = _first_str(data.get("contents"), data.get("ownerAuthContents")) + if contents: + out["signature"] = contents + out["following_count"] = _first_int(data.get("upFollowNum"), data.get("up_follow_num")) + out["follower_count"] = _first_int(data.get("subscribeNum"), data.get("subscribe_num")) + # Video play count — shown as「播放」on Douyu cards. + out["total_favorited"] = _first_int(data.get("playCount"), data.get("play_count")) + avatar = _first_str(data.get("avatar")) + if avatar: + out["avatar_url"] = avatar.replace(r"\/", "/") + return out + + +def fetch_douyu_author_profile( + opener: urllib.request.OpenerDirector, profile_url: str +) -> dict: + """Fetch Douyu author / author-video page and parse ``window.$DATA``.""" + m = DOUYU_AUTHOR_HASH_RE.search(profile_url) + if not m: + raise ValueError(f"not a douyu author url: {profile_url!r}") + hash_id = m.group(1) + # Prefer /author/; fall back to /author-video/ if $DATA is missing. + errors: list[str] = [] + for path in (f"author/{hash_id}", f"author-video/{hash_id}"): + page_url = f"https://v.douyu.com/{path}" + try: + _, html = _get(opener, page_url, referer="https://v.douyu.com/") + except (urllib.error.URLError, TimeoutError, OSError) as e: + errors.append(f"{path}: {e}") + continue + data_m = DOUYU_DATA_RE.search(html) + if not data_m: + errors.append(f"{path}: no window.$DATA") + continue + profile = extract_profile_from_douyu_data( + parse_douyu_dollar_data(data_m.group(1)) + ) + if not any( + profile.get(k) is not None + for k in ("nickname", "follower_count", "avatar_url", "unique_id") + ): + errors.append(f"{path}: parsed empty") + continue + return profile + raise ValueError("; ".join(errors) or "douyu author page failed") + + +def extract_douyu_up_id_from_room_html(html: str) -> str | None: + """Pull author ``up_id`` hash embedded in the live-room page HTML.""" + m = DOUYU_UP_ID_RE.search(html or "") + return m.group(1) if m else None + + +def resolve_douyu_author_url_from_room( + opener: urllib.request.OpenerDirector, room_id: str +) -> str: + """Fetch ``www.douyu.com/`` and build the author homepage URL.""" + page_url = f"https://www.douyu.com/{room_id}" + _, html = _get(opener, page_url, referer="https://www.douyu.com/") + up_id = extract_douyu_up_id_from_room_html(html) + if not up_id: + raise ValueError(f"douyu room {room_id}: no up_id in HTML") + return f"https://v.douyu.com/author/{up_id}" + + +def fetch_douyu_room_profile(room_id: str) -> dict: + """Fallback enrichment from live-room ``betard`` (no fan counts).""" + payload = http_json(DOUYU_BETARD_URL.format(room_id=room_id), timeout=20) + if not isinstance(payload, dict): + raise ValueError("douyu betard returned non-object") + room = payload.get("room") + if not isinstance(room, dict): + raise ValueError("douyu betard returned no room") + out: dict[str, Any] = { + "unique_id": str(room_id), + "live_url": f"https://www.douyu.com/{room_id}", + } + nick = _first_str(room.get("nickname"), room.get("owner_name")) + if nick: + out["nickname"] = nick + details = _first_str(room.get("show_details"), room.get("room_name")) + if details: + out["signature"] = details + avatar = room.get("avatar") + avatar_url = None + if isinstance(avatar, dict): + avatar_url = _first_str(avatar.get("big"), avatar.get("middle"), avatar.get("small")) + elif isinstance(avatar, str): + avatar_url = avatar + if not avatar_url: + avatar_url = _first_str(room.get("owner_avatar"), room.get("avatar_mid")) + if avatar_url: + out["avatar_url"] = avatar_url + if not out.get("nickname") and not out.get("avatar_url"): + raise ValueError("douyu betard parsed empty") + return out + + +def fetch_douyu_profile( + opener: urllib.request.OpenerDirector, row: dict +) -> dict: + """Enrich a Douyu streamer from author URL, room→up_id, else betard.""" + profile_url = str(row.get("profile_url") or "").strip() + live_url = str(row.get("live_url") or "").strip() + if DOUYU_AUTHOR_HASH_RE.search(profile_url): + return fetch_douyu_author_profile(opener, profile_url) + room_id = None + for candidate in (profile_url, live_url): + m = DOUYU_ROOM_RE.search(candidate) + if m: + room_id = m.group(1) + break + if not room_id: + raise ValueError("douyu row needs author profile_url or room live_url") + try: + author_url = resolve_douyu_author_url_from_room(opener, room_id) + return fetch_douyu_author_profile(opener, author_url) + except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e: + print(f" douyu room→author failed ({e}); betard fallback", flush=True) + return fetch_douyu_room_profile(room_id) + + def fetch_douyin_profile( opener: urllib.request.OpenerDirector, profile_url: str ) -> dict: @@ -382,6 +561,15 @@ def merge_profile(row: dict, profile: dict, *, streamer_id: str) -> None: if val is None or val == "": continue row[key] = val + # Fill missing live/profile URLs from platform enrichment; never wipe seeds. + for key in ("live_url", "profile_url"): + val = profile.get(key) + if isinstance(val, str) and val and not str(row.get(key) or "").strip(): + row[key] = val + # Prefer canonical Douyu author homepage when enrichment found one. + prof = profile.get("profile_url") + if isinstance(prof, str) and DOUYU_AUTHOR_HASH_RE.search(prof): + row["profile_url"] = prof avatar_url = profile.get("avatar_url") if isinstance(avatar_url, str) and avatar_url: dest = AVATAR_DIR / f"{streamer_id}.jpg" @@ -408,18 +596,25 @@ def enrich_streamers( if ids is not None and sid not in ids: continue platform = str(row.get("platform") or "").strip().lower() - if platform != "douyin": - print(f"skip {sid}: platform={platform!r} (only douyin supported)", flush=True) + if platform not in ("douyin", "douyu"): + print( + f"skip {sid}: platform={platform!r} " + f"(supported: douyin, douyu)", + flush=True, + ) skip += 1 continue - profile_url = str(row.get("profile_url") or "").strip() - if not profile_url: - print(f"skip {sid}: missing profile_url", flush=True) - skip += 1 - continue - print(f"fetching {sid} ...", flush=True) + print(f"fetching {sid} ({platform}) ...", flush=True) try: - profile = fetch_douyin_profile(opener, profile_url) + if platform == "douyin": + profile_url = str(row.get("profile_url") or "").strip() + if not profile_url: + print(f"skip {sid}: missing profile_url", flush=True) + skip += 1 + continue + profile = fetch_douyin_profile(opener, profile_url) + else: + profile = fetch_douyu_profile(opener, row) merge_profile(row, profile, streamer_id=sid) nick = row.get("nickname") or "?" print( @@ -434,7 +629,7 @@ def enrich_streamers( time.sleep(0.8) if ok > 0: payload["fetched_at"] = _now_iso() - payload["source"] = payload.get("source") or "manual+douyin" + payload["source"] = payload.get("source") or "manual+douyin+douyu" meta = payload.get("platform_meta") if not isinstance(meta, dict): meta = {} @@ -442,13 +637,19 @@ def enrich_streamers( "douyin", {"label_zh": "抖音", "icon": "ui-icon/platform_douyin.png"}, ) + meta.setdefault( + "douyu", + {"label_zh": "斗鱼", "icon": "ui-icon/platform_douyu.png"}, + ) payload["platform_meta"] = meta _ = PROFILE_KEYS return ok, skip, fail def main() -> int: - ap = argparse.ArgumentParser(description="Enrich streamers.json from Douyin profiles") + ap = argparse.ArgumentParser( + description="Enrich streamers.json from Douyin / Douyu profiles" + ) ap.add_argument("--out", type=Path, default=OUT) ap.add_argument( "--ids", diff --git a/web/frontend/app.js b/web/frontend/app.js index b20ab72..bd10d23 100644 --- a/web/frontend/app.js +++ b/web/frontend/app.js @@ -4364,7 +4364,14 @@ function buildStreamerMetaLine(row) { const parts = []; if (uniqueId) { parts.push({ - label: platform === "douyin" ? "抖音号" : platform === "bilibili" ? "UID" : "账号", + label: + platform === "douyin" + ? "抖音号" + : platform === "bilibili" + ? "UID" + : platform === "douyu" + ? "房间号" + : "账号", value: String(uniqueId), account: true, }); @@ -4373,6 +4380,13 @@ function buildStreamerMetaLine(row) { if (Number.isFinite(Number(row.following_count))) { parts.push({ label: "关注", value: formatStreamerCount(row.following_count) }); } + } else if (platform === "douyu") { + if (Number.isFinite(Number(row.following_count))) { + parts.push({ label: "关注", value: formatStreamerCount(row.following_count) }); + } + if (Number.isFinite(Number(row.total_favorited))) { + parts.push({ label: "播放", value: formatStreamerCount(row.total_favorited) }); + } } else if (Number.isFinite(Number(row.total_favorited))) { parts.push({ label: "获赞", value: formatStreamerCount(row.total_favorited) }); } diff --git a/web/frontend/config.js b/web/frontend/config.js index 276c649..36ae99a 100644 --- a/web/frontend/config.js +++ b/web/frontend/config.js @@ -1,5 +1,5 @@ /* Local defaults; production export overwrites via export_relations_site.py. */ -var SITE_VERSION = "0.5.112"; +var SITE_VERSION = "0.5.114"; var ABILITY_VIDEO_BASE = ""; var STATIC_ASSET_BASE = ""; diff --git a/web/frontend/functions/api/live-status.js b/web/frontend/functions/api/live-status.js index ee43df2..2ac65fb 100644 --- a/web/frontend/functions/api/live-status.js +++ b/web/frontend/functions/api/live-status.js @@ -10,6 +10,8 @@ * Probe logic is a JS port of fetch_streamer_live.py: * - Bilibili: api.live.bilibili.com Room/get_info; data.live_status === 1 is * live (0 offline, 2 replay counts as offline). + * - Douyu: www.douyu.com/betard/; room.show_status === 1 is live + * (2 offline); room.videoLoop === 1 (carousel) counts as offline. * - Douyin: warm up cookies (www.douyin.com + live.douyin.com), then GET * live.douyin.com/ with a browser UA and parse the escaped JSON in * the SSR pace chunks: roomStore.roomInfo.room.status (2 live / 4 offline); @@ -44,6 +46,7 @@ const DOUYIN_HOME = "https://www.douyin.com/"; const DOUYIN_LIVE_HOME = "https://live.douyin.com/"; const BILIBILI_INFO_URL = "https://api.live.bilibili.com/room/v1/Room/get_info?room_id="; +const DOUYU_BETARD_URL = "https://www.douyu.com/betard/"; // Escaped JSON inside the SSR pace chunks: \"roomStore\":{\"roomInfo\":{\"room\":{ const DOUYIN_ROOMSTORE_RE = /\\"roomStore\\":\s*\{\\"roomInfo\\":\s*\{\\"room\\":\s*\{/; @@ -162,7 +165,25 @@ export async function probeBilibiliRoom(roomId) { return data.live_status === 1; } -/** First path segment of the live room URL (douyin web_rid / bilibili room id). */ +/** show_status: 1 live, 2 offline; videoLoop carousel counts as offline. */ +export async function probeDouyuRoom(roomId) { + const res = await fetchWithTimeout(DOUYU_BETARD_URL + roomId, { + headers: { + "User-Agent": BROWSER_UA, + Accept: "application/json", + Referer: "https://www.douyu.com/", + }, + }); + const payload = await res.json(); + const room = payload && payload.room; + if (!room || typeof room !== "object") { + throw new Error("douyu betard returned no room"); + } + if (Number(room.videoLoop || 0) === 1) return false; + return Number(room.show_status) === 1; +} + +/** First path segment of the live room URL (douyin/bilibili/douyu room id). */ export function roomRefFromUrl(liveUrl) { let path = ""; try { @@ -180,6 +201,7 @@ export function livePlatformFromUrl(liveUrl, fallback = "") { const host = new URL(String(liveUrl).trim()).hostname.toLowerCase(); if (host.includes("bilibili.com")) return "bilibili"; if (host.includes("douyin.com")) return "douyin"; + if (host.includes("douyu.com")) return "douyu"; } catch { /* keep fallback */ } @@ -245,6 +267,7 @@ async function probeAll(targets) { let isLive; if (t.platform === "douyin") isLive = await probeDouyinRoom(t.ref, jar); else if (t.platform === "bilibili") isLive = await probeBilibiliRoom(t.ref); + else if (t.platform === "douyu") isLive = await probeDouyuRoom(t.ref); else throw new Error(`unsupported platform ${t.platform}`); results.set(t.id, { is_live: isLive }); } catch (err) { diff --git a/web/frontend/index.html b/web/frontend/index.html index 1a8a419..b5fa8c5 100644 --- a/web/frontend/index.html +++ b/web/frontend/index.html @@ -5,8 +5,8 @@ DOTA2 上分帝 - - + +

DOTA2 上分帝

@@ -178,8 +178,8 @@
- - - + + + diff --git a/web/tests/test_refresh_reliability.py b/web/tests/test_refresh_reliability.py index 50cf880..193cf2f 100644 --- a/web/tests/test_refresh_reliability.py +++ b/web/tests/test_refresh_reliability.py @@ -17,6 +17,12 @@ sys.path.insert(0, str(ROOT / "web")) from web import notify_site_traffic, refresh_web from web.fetch_stratz_meta import preserve_failed_meta_brackets from web.fetch_streamer_live import probe_streamers +from web.fetch_streamers import ( + DOUYU_AUTHOR_HASH_RE, + extract_douyu_up_id_from_room_html, + extract_profile_from_douyu_data, + parse_douyu_dollar_data, +) from web.refresh_cache import restore_patches, restore_streamers from shared.http_utils import write_json_atomic from shared.paths import HEROES_JSON, RELATIONS_JSON, WEB_FRONTEND @@ -260,6 +266,65 @@ class LiveProbeFallbackTests(unittest.TestCase): self.assertFalse(row["is_live"]) self.assertNotIn("live_probed_at", row) + def test_douyu_probe_path_writes_live_flag(self) -> None: + payload = { + "streamers": [ + { + "id": "gouhuang", + "platform": "douyu", + "live_url": "https://www.douyu.com/957090", + } + ] + } + with patch("web.fetch_streamer_live.probe_douyu", return_value=True): + live, offline, fail = probe_streamers(payload) + self.assertEqual((live, offline, fail), (1, 0, 0)) + self.assertTrue(payload["streamers"][0]["is_live"]) + self.assertIn("live_probed_at", payload["streamers"][0]) + + +class DouyuProfileParseTests(unittest.TestCase): + def test_author_and_author_video_hashes(self) -> None: + self.assertEqual( + DOUYU_AUTHOR_HASH_RE.search( + "https://v.douyu.com/author/JPw9YOLKlw5X" + ).group(1), + "JPw9YOLKlw5X", + ) + self.assertEqual( + DOUYU_AUTHOR_HASH_RE.search( + "https://v.douyu.com/author-video/JPw9YOLKlw5X" + ).group(1), + "JPw9YOLKlw5X", + ) + + def test_dollar_data_maps_fan_fields(self) -> None: + blob = ( + '{type:"0",uid:"13506421",upId:"JPw9YOLKlw5X",upFollowNum:"16",' + 'subscribeNum:"465781",playCount:"26074077",' + 'avatar:"https://example.com/a.jpg",' + 'name: "18yearsold\\u5929\\u6b8b\\u5c11\\u5e74K9",' + 'contents: "",roomId:"235520",roomName: "k9 room"}' + ) + profile = extract_profile_from_douyu_data(parse_douyu_dollar_data(blob)) + self.assertEqual(profile["nickname"], "18yearsold天残少年K9") + self.assertEqual(profile["unique_id"], "235520") + self.assertEqual(profile["following_count"], 16) + self.assertEqual(profile["follower_count"], 465781) + self.assertEqual(profile["total_favorited"], 26074077) + self.assertEqual(profile["profile_url"], "https://v.douyu.com/author/JPw9YOLKlw5X") + self.assertEqual(profile["live_url"], "https://www.douyu.com/235520") + self.assertNotIn("signature", profile) + + def test_room_html_up_id_extraction(self) -> None: + plain = '{"rid":957090,"up_id":"EqAvg1lQD75L","ban_display":0}' + escaped = ( + r'{\"rid\":957090,\"up_id\":\"EqAvg1lQD75L\",\"ban_display\":0}' + ) + self.assertEqual(extract_douyu_up_id_from_room_html(plain), "EqAvg1lQD75L") + self.assertEqual(extract_douyu_up_id_from_room_html(escaped), "EqAvg1lQD75L") + self.assertIsNone(extract_douyu_up_id_from_room_html("")) + class NotifySummaryTests(unittest.TestCase): def test_skipped_summary_is_labeled(self) -> None: