v0.5.114: add Douyu streamers with fan enrichment and live probe.

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-07-30 02:42:47 +08:00
parent fdd926e9bb
commit 38f46ad2ea
17 changed files with 477 additions and 38 deletions
@@ -88,9 +88,9 @@ climperor/
 | `web/fetch_hero_matches.py` | 拉取同英雄近期比赛 + 终局出装/加点 → `web/data/hero_matches.json`（`--source league\|public\|both`；合并后保留最近 N 场胜局，默认 10；天梯需传奇及以上；公开列表过滤 bot/Turbo、仅 ranked lobby；`--public-region china` 优先国服；`--workers` 详情并发 + 凑满即停；可选 `OPENDOTA_API_KEY`；增量补缺失/不足 N 场；`--enrich-item-times` 补购买时间；**仅上分帝 Web**；勿进 recommend） |
 | `web/fetch_pro_matches.py` | 按明星名单拉近期联赛/锦标赛对局 → `web/data/pro_matches.json`（默认读 `web/data/pro_player_watchlist.json`；`--include-pubs` 另拉天梯 lobby 7；每种 lobby 各保留 `--limit` 场；`--players` 覆盖整文件；`--all-pros` 从 `/proPlayers` 盲抽；含终局出装/加点；进 `refresh_web` daily 且带 `--include-pubs`；**仅上分帝 Web「比赛」**；勿进 recommend） |
 | `web/fetch_leaderboards.py` | 拉取 Valve Immortal 四区榜 Top100 → `web/data/leaderboards.json`（**仅上分帝 Web「排行」**；无 MMR/account_id；勿进 recommend） |
-| `web/fetch_streamers.py` | 从抖音主页补全 `web/data/streamers.json` 的昵称/签名/关注粉丝获赞/头像（手工名单 + 直播间/主页 URL；支持 `v.douyin.com` 短链；失败保留旧值；**不**探测开播（由 `fetch_streamer_live.py` 负责）；**仅上分帝 Web「主播」**；勿进 recommend；进 `refresh_web` daily） |
+| `web/fetch_streamers.py` | 从抖音 / 斗鱼主页补全 `web/data/streamers.json` 的昵称/签名/关注粉丝获赞(播放)/头像（手工名单 + 直播间/主页 URL；抖音支持 `v.douyin.com` 短链；斗鱼优先 `v.douyu.com/author/<hash>`（兼容 `author-video`）的 `$DATA`，仅房间号时从直播间 HTML 解析 `up_id` 再拉作者页，失败才回退 `betard`；失败保留旧值；**不**探测开播（由 `fetch_streamer_live.py` 负责）；**仅上分帝 Web「主播」**；勿进 recommend；进 `refresh_web` daily） |
-| `web/fetch_streamer_live.py` | 探测主播真实在播状态回写 `web/data/streamers.json` 的 `is_live`/`live_probed_at`：抖音解析直播间 SSR 页 `roomStore.roomInfo.room.status`（2 在播 / 4 下播；预热 cookie + ~1s 间隔；web_rid 校验），B 站走 `Room/get_info`（`live_status==1` 在播，轮播算下播）；探测失败清为 `is_live:false` 并去掉 `live_probed_at`（与 `/api/live-status` 一致，不沿用旧直播中）、始终 exit 0；`--ids a,b` 限范围、`--dry-run` 只打印；仅 Web；进 `refresh_web` daily（角标以访问触发的 live API 为准，daily 仅作 data.json 兜底） |
+| `web/fetch_streamer_live.py` | 探测主播真实在播状态回写 `web/data/streamers.json` 的 `is_live`/`live_probed_at`：抖音解析直播间 SSR 页 `roomStore.roomInfo.room.status`（2 在播 / 4 下播；预热 cookie + ~1s 间隔；web_rid 校验），B 站走 `Room/get_info`（`live_status==1` 在播，轮播算下播），斗鱼走 `betard/{room_id}`（`show_status==1` 在播，`videoLoop==1` 轮播算下播）；探测失败清为 `is_live:false` 并去掉 `live_probed_at`（与 `/api/live-status` 一致，不沿用旧直播中）、始终 exit 0；`--ids a,b` 限范围、`--dry-run` 只打印；仅 Web；进 `refresh_web` daily（角标以访问触发的 live API 为准，daily 仅作 data.json 兜底） |
-| `web/frontend/functions/api/live-status.js` | Pages Function `GET /api/live-status`：访问触发的在播探测（逻辑同 `fetch_streamer_live.py`），读 `data.json` 的 `streamers.streamers`；Cache API 固定键 + isolate 内 in-flight 合并（5 分钟新鲜窗口，**无 KV**）；抖音从数据中心 IP 失败属预期 → 失败主播一律 `is_live:false` + `stale:true`（**不**沿用旧的直播中）；全失败回 `stale-override` 空角标表或 `error`，永不 500；导出时拷贝 `functions/`；**部署须 `cwd=dist` 跑 wrangler**（Functions 相对 cwd 解析） |
+| `web/frontend/functions/api/live-status.js` | Pages Function `GET /api/live-status`：访问触发的在播探测（逻辑同 `fetch_streamer_live.py`，含抖音 / B 站 / 斗鱼），读 `data.json` 的 `streamers.streamers`；Cache API 固定键 + isolate 内 in-flight 合并（5 分钟新鲜窗口，**无 KV**）；抖音从数据中心 IP 失败属预期 → 失败主播一律 `is_live:false` + `stale:true`（**不**沿用旧的直播中）；全失败回 `stale-override` 空角标表或 `error`，永不 500；导出时拷贝 `functions/`；**部署须 `cwd=dist` 跑 wrangler**（Functions 相对 cwd 解析） |
 | `web/frontend/functions/api/mobile-demand.js` | Pages Function `GET\|POST /api/mobile-demand`：移动端「催更」需求计数（Cache API 存 `count`，**无 KV**；边缘竞态/驱逐可能少计或重置）；本机 `serve_relations.py` 写 `web/.refresh/mobile_demand.json`；前端 `mobile-gate.js` 用 UA 识别手机/平板并拦截，`localStorage` 同设备只 POST 一次 |
 | `web/frontend/mobile-gate.js` | 移动端门禁（`<head>` 早载）：`html.mobile-client` + 催更按钮；设 `window.__CLIMPEROR_MOBILE__` 供 `app.js` 跳过桌面 boot |
 | `web/fetch_item_shop.py` | 官网商店 11 列目录（dota2.com.cn/itemscategory）+ 合成图 → `web/data/item_shop.json` + 图标 |
@@ -109,7 +109,7 @@ climperor/
 | `web/data/pro_player_watchlist.json` | 明星选手 OpenDota `account_id` 手工名单（对照 Dotabuff / Liquipedia；含现役战队席位与昔日国服明星；外号：查理斯→Chalice，Somnus 即 Maybe；CN/EU/SA 等）；`fetch_pro_matches.py` 默认只拉此名单；队名可过期，以 id 为准；**不**进 recommend |
 | `web/data/pro_matches.json` | 明星选手近期联赛/锦标赛（可选天梯）对局（`by_pro`/`by_hero` + 终局出装/加点 + `lobby_type`/`origin`；由 `fetch_pro_matches.py` 生成；Web 顶层「比赛」侧栏筛「全部 / 职业 / 国服」+ 选手，与英雄详情「近期比赛」合并展示；进 daily；**不**参与局内推荐） |
 | `web/data/leaderboards.json` | Valve Immortal 四区 Top100（`china`/`europe`/`americas`/`se_asia`；仅排名+昵称等；由 `fetch_leaderboards.py` 生成；Web「排行」选手榜只读；**不**参与局内推荐） |
-| `web/data/streamers.json` | 主播目录（手工 `live_url`/profile URL + 常用英雄 + 可选精选视频 `video`/`video_title` + 可选 `video_poster`/`video_fit`/`video_crop`/`video_aspect`；抖音/B 站 profile 补全；`platform_meta`；Web 卡片：头像行 = 头像 \| 昵称+账号/获赞粉丝 \|「关注」，签名（`signature`）独立全宽行（最长 3 行）；在播时粉环 +「直播」角标叠在环底（抖音式 `bottom:-6px`，无间距）；列表排序：先 `is_live` 再粉丝数降序；视口分档滚播（远处不拉、近处 metadata、中部 `canplay` 且单路 `preload=auto`）+ 同名 JPG 封面占位；有 `live_url` 时点头像进直播间；**「直播」角标只信本轮成功探测**（线上 `/api/live-status` 边缘缓存 5 分钟；本地 `serve_relations` 同逻辑缓存 60s；`stale`/失败不显示角标；`data.json` daily 仅作首屏兜底直至接口返回；`live_url` 仅作点击入口）；由 `fetch_streamers.py` 补全；Web「主播」只读；**不**参与局内推荐） |
+| `web/data/streamers.json` | 主播目录（手工 `live_url`/profile URL + 常用英雄 + 可选精选视频 `video`/`video_title` + 可选 `video_poster`/`video_fit`/`video_crop`/`video_aspect`；抖音 profile 补全；平台含抖音 / B 站 / 斗鱼；`platform_meta`；Web 卡片：头像行 = 头像 \| 昵称+账号/获赞粉丝 \|「关注」，签名（`signature`）独立全宽行（最长 3 行）；在播时粉环 +「直播」角标叠在环底（抖音式 `bottom:-6px`，无间距）；列表排序：先 `is_live` 再粉丝数降序；视口分档滚播（远处不拉、近处 metadata、中部 `canplay` 且单路 `preload=auto`）+ 同名 JPG 封面占位；有 `live_url` 时点头像进直播间；**「直播」角标只信本轮成功探测**（线上 `/api/live-status` 边缘缓存 5 分钟；本地 `serve_relations` 同逻辑缓存 60s；`stale`/失败不显示角标；`data.json` daily 仅作首屏兜底直至接口返回；`live_url` 仅作点击入口）；由 `fetch_streamers.py` 补全；Web「主播」只读；**不**参与局内推荐） |
 | `web/data/stratz_hero_meta.json` | STRATZ 各勋章段位近 N 周 pick/win（`weeks`/`latest`）+ **同段位最近 1 周分路**（`winWeek`+`positionIds`）+ `meta_board`（由 `fetch_stratz_meta.py` 生成；Web 英雄详情「走势」优先 + 顶层「走势」`#/trends` 近 N 周榜；**不**参与局内推荐） |
 | `web/data/stratz_matchup_tops.json` | STRATZ 对位/协同 Top（counters/countered/synergies；由 `fetch_stratz_meta.py` 生成；全局聚合 + `scope`/`stale`/`fetched_at`；Web「对位」Tab；与定性 `relations.json` 分开展示；**不**参与局内推荐） |
 | `web/data/item_shop.json` | 商店 11 列目录（官网 basic/upgrade；由 `fetch_item_shop.py` 生成；物品页只读） |
@@ -163,7 +163,7 @@ Dota 2 GSI → pc/gsi_watch.py (:3223)
 - **合规**：仅 GSI + 屏幕截图；**禁止**读进程内存、注入、绕过 VAC。
 - **GSI 范围**：普通玩家视角拿不到双方 pick；GSI 只作阶段触发、`team_slot`、本人 `hero`、本机 `accountid`/`steamid`。本人顶栏槽位**只信** GSI，不用截屏名字亮度猜测。
- **克制 / 搭档数据**：机制克制/搭档只用定性边存 `shared/data/relations.json`（克制有向 + 搭档无向 + 理由），**不要**用胜率/场次表达机制克制，也**不要**写入 `shared/data/heroes.json`。Web 英雄页三视图：克制 / 被克制 / 搭档。OpenDota 段位胜率/场次单独存 `web/data/hero_stats.json`（近约 7 天窗；上场率 = 出场/(Σ出场/10)；冠绝样本过小时与超凡合并），**仅**上分帝 Web 英雄详情「走势」**无 STRATZ 时兜底**。STRATZ 周胜率/分路/`meta_board` 存 `web/data/stratz_hero_meta.json`（三卡与分路均取 **当前选中勋章 · 最近 1 周** `winWeek`；近 8 周列表从新到旧；分路在 8 周列表上方），对位 Top 存 `web/data/stratz_matchup_tops.json`（英雄详情「走势」与顶层「走势」榜 `#/trends` 优先用 STRATZ；「对位」Tab 只读），**禁止**合并进 relations/heroes，禁止 `pc/recommend.py` 读取。同英雄近期比赛出装/加点单独存 `web/data/hero_matches.json`，**仅** Web「近期比赛」Tab，禁止进 recommend。Valve Immortal 四区榜单独存 `web/data/leaderboards.json`，**仅** Web「排行」页选手榜，禁止进 recommend。主播目录单独存 `web/data/streamers.json`（手工名单 + 抖音 profile 补全；首版仅抖音），**仅** Web 顶层「主播」与英雄详情「主播」Tab，禁止进 recommend；由 `refresh_web` daily 软失败刷新粉丝等字段（不阻断整档）。
+- **克制 / 搭档数据**：机制克制/搭档只用定性边存 `shared/data/relations.json`（克制有向 + 搭档无向 + 理由），**不要**用胜率/场次表达机制克制，也**不要**写入 `shared/data/heroes.json`。Web 英雄页三视图：克制 / 被克制 / 搭档。OpenDota 段位胜率/场次单独存 `web/data/hero_stats.json`（近约 7 天窗；上场率 = 出场/(Σ出场/10)；冠绝样本过小时与超凡合并），**仅**上分帝 Web 英雄详情「走势」**无 STRATZ 时兜底**。STRATZ 周胜率/分路/`meta_board` 存 `web/data/stratz_hero_meta.json`（三卡与分路均取 **当前选中勋章 · 最近 1 周** `winWeek`；近 8 周列表从新到旧；分路在 8 周列表上方），对位 Top 存 `web/data/stratz_matchup_tops.json`（英雄详情「走势」与顶层「走势」榜 `#/trends` 优先用 STRATZ；「对位」Tab 只读），**禁止**合并进 relations/heroes，禁止 `pc/recommend.py` 读取。同英雄近期比赛出装/加点单独存 `web/data/hero_matches.json`，**仅** Web「近期比赛」Tab，禁止进 recommend。Valve Immortal 四区榜单独存 `web/data/leaderboards.json`，**仅** Web「排行」页选手榜，禁止进 recommend。主播目录单独存 `web/data/streamers.json`（手工名单 + 抖音 profile 补全；平台含抖音 / B 站 / 斗鱼），**仅** Web 顶层「主播」与英雄详情「主播」Tab，禁止进 recommend；由 `refresh_web` daily 软失败刷新粉丝等字段（不阻断整档）。
 - **装备机制 / 怕的装备**：标签与技能汇总来自 Valve/OpenDota 自动抽取 + `item_tag_overrides.json`；`item_fears.py` 规则映射到英雄弱点。`fetch_item_counter_stats.py` 的对阵购买率/胜率差仅作为观测证据：须减去同装备全局基线，对全部英雄的已有机制候选只允许小幅调序；统计发现的新组合即使同时满足 `games≥100`、购买率提升 `≥3pp`、条件胜率差 `≥1.5pp`、两项双比例检验 `z≥1.96`，仍须确认机制成立后手工写入 overrides，禁止直接把高相关当因果克制。大段技能文案只进 `web/data/hero_abilities.json` / `items_meta.json`，**不要**塞进 `heroes.json`。本阶段仅上分帝 Web 展示，**不对局内出装推荐**。核心装「使用率」为 `hero_items` 列表内相对热度归一化，非绝对出场率、无段位维度。技能/物品「施加」类 tags（机制查询页）与技能 `dispellable`（效果能否被驱散）严格区分；**禁止**写入 recommend / relations。
 - **机制查询页**：顶层 `#/mechanics[/{effect}]`；侧栏弱/强驱散 + 核心控制；结果为施加该效果的技能与物品。数据边界：仅 Web。
 - **模板策略**：只维护 CDN 层。皮肤问题用会话策略（选人可改判、决策 `allow_revise=False`、best 帧偏 HERO_SELECTION），不要为皮肤加模板库，也不要复活 real 双层库。
@@ -4,6 +4,13 @@
 ## [Unreleased]
 ## [0.5.114] - 2026-07-30
 ### Added
 - 主播资料补全支持斗鱼：`v.douyu.com/author/<hash>`（兼容 `author-video`）拉粉丝/关注/播放/头像；仅有房间号时从直播间 HTML 解析 `up_id` 再拉作者页（失败才回退 `betard`）。斗鱼卡片展示「关注 / 播放 / 粉丝」。
 - 主播页收录斗鱼「踏上征途167（狗皇）」「天残少年K9」「叶子长青K」、抖音「浙大小鱼王—dota2」与 B 站「Dota2绝中绝」；开播探测支持斗鱼（`betard`）。
 ## [0.5.112] - 2026-07-30
 ### Changed
@@ -1,6 +1,6 @@
 {
-  "fetched_at": "2026-07-28T14:20:00+00:00",
+  "fetched_at": "2026-07-29T18:37:21.491922+00:00",
-  "source": "manual+douyin+bilibili",
+  "source": "manual+douyin+bilibili+douyu",
  "platform_meta": {
    "douyin": {
      "label_zh": "抖音",
@@ -9,6 +9,10 @@
    "bilibili": {
      "label_zh": "哔哩哔哩",
      "icon": "ui-icon/platform_bilibili.png"
    },
    "douyu": {
      "label_zh": "斗鱼",
      "icon": "ui-icon/platform_douyu.png"
    }
  },
  "streamers": [
@@ -480,6 +484,102 @@
      "profile_fetched_at": "2026-07-29T09:54:00+00:00",
      "is_live": true,
      "live_probed_at": "2026-07-29T09:54:00+00:00"
    },
    {
      "id": "gouhuang",
      "platform": "douyu",
      "live_url": "https://www.douyu.com/957090",
      "profile_url": "https://v.douyu.com/author/EqAvg1lQD75L",
      "tagline": "米波绝活 · 狗皇",
      "heroes": [
        "meepo",
        "broodmother",
        "alchemist"
      ],
      "nickname": "踏上征途167",
      "unique_id": "957090",
      "signature": "娱乐DOTA 轻松氛围",
      "avatar": "streamer_avatars/gouhuang.jpg",
      "is_live": false,
      "live_probed_at": "2026-07-29T18:16:27.171378+00:00",
      "profile_fetched_at": "2026-07-29T18:37:18.228078+00:00",
      "following_count": 868,
      "follower_count": 280201,
      "total_favorited": 9673470
    },
    {
      "id": "juezhongjue",
      "platform": "bilibili",
      "live_url": "https://live.bilibili.com/1747114599",
      "profile_url": "https://space.bilibili.com/520653835",
      "tagline": "绝活打野炼金",
      "heroes": [
        "alchemist"
      ],
      "nickname": "Dota2绝中绝",
      "unique_id": "520653835",
      "signature": "绝活打野炼金。鄙人不擅长对线对线，但极其擅长与野怪掰头！",
      "follower_count": 433,
      "avatar": "streamer_avatars/juezhongjue.jpg",
      "is_live": false,
      "live_probed_at": "2026-07-29T18:08:20.096701+00:00"
    },
    {
      "id": "k9",
      "platform": "douyu",
      "live_url": "https://www.douyu.com/235520",
      "profile_url": "https://v.douyu.com/author/JPw9YOLKlw5X",
      "tagline": "绝活卡尔",
      "heroes": [
        "invoker"
      ],
      "nickname": "18yearsold天残少年K9",
      "unique_id": "235520",
      "signature": "k9：午夜牢车教室！.",
      "avatar": "streamer_avatars/k9.jpg",
      "is_live": true,
      "live_probed_at": "2026-07-29T18:16:27.995333+00:00",
      "following_count": 16,
      "follower_count": 465781,
      "total_favorited": 26074078,
      "profile_fetched_at": "2026-07-29T18:23:59.723287+00:00"
    },
    {
      "id": "yezi",
      "platform": "douyu",
      "live_url": "https://www.douyu.com/246195",
      "profile_url": "https://v.douyu.com/author/Kqy70jkrpdXG",
      "tagline": "绝活火猫",
      "heroes": [
        "ember_spirit"
      ],
      "nickname": "叶子长青K",
      "unique_id": "246195",
      "signature": "QQ:394867894  网易云搜用户 淡然听听歌",
      "avatar": "streamer_avatars/yezi.jpg",
      "is_live": false,
      "live_probed_at": "2026-07-29T18:16:28.416689+00:00",
      "profile_fetched_at": "2026-07-29T18:37:20.691635+00:00",
      "following_count": 134,
      "follower_count": 286391,
      "total_favorited": 6336746
    },
    {
      "id": "zhedaxiaoyuwang",
      "platform": "douyin",
      "profile_url": "https://v.douyin.com/tSe_vsZ8r7o/",
      "tagline": "小鱼人",
      "heroes": [
        "slark"
      ],
      "nickname": "浙大小鱼王—dota2",
      "unique_id": "89748079578",
      "signature": "本硕985，浙大在读 9700冲万分",
      "following_count": 52,
      "follower_count": 1802,
      "total_favorited": 842,
      "avatar": "streamer_avatars/zhedaxiaoyuwang.jpg",
      "profile_fetched_at": "2026-07-29T18:26:00+00:00"
    }
  ]
 }
@@ -52,7 +52,7 @@ from shared.paths import (
 from serve_relations import WEB_DIR, build_payload
-SITE_VERSION = "0.5.112"
+SITE_VERSION = "0.5.114"
 DEFAULT_OSS_BASE = "https://climperor.oss-cn-shanghai.aliyuncs.com"
@@ -13,6 +13,9 @@ Approach (verified 2026-07):
 - Bilibili: public API ``api.live.bilibili.com/room/v1/Room/get_info`` with the
  numeric room id taken from the ``live_url`` path; ``data.live_status == 1``
  means live (0 offline, 2 replay — replay is treated as offline). No login.
 - Douyu: public ``www.douyu.com/betard/{room_id}``; ``room.show_status == 1``
  means live, ``== 2`` offline; ``room.videoLoop == 1`` (carousel) counts as
  offline.
 - Douyin: one shared cookie session is warmed up (www.douyin.com +
  live.douyin.com), then per room we GET ``live.douyin.com/{web_rid}`` with a
  browser UA (the numeric ``live_url`` path segment is the ``web_rid``). The
@@ -65,10 +68,12 @@ TIMEOUT = 20
 # Douyin rate-limits aggressively; keep ~1s spacing between its requests.
 DOUYIN_SPACING = 1.0
 BILIBILI_SPACING = 0.5
 DOUYU_SPACING = 0.3
 DOUYIN_HOME = "https://www.douyin.com/"
 DOUYIN_LIVE_HOME = "https://live.douyin.com/"
 BILIBILI_INFO_URL = "https://api.live.bilibili.com/room/v1/Room/get_info?room_id={room_id}"
 DOUYU_BETARD_URL = "https://www.douyu.com/betard/{room_id}"
 # Escaped JSON inside the SSR pace chunks: \"roomStore\":{\"roomInfo\":{\"room\":{
 DOUYIN_ROOMSTORE_RE = re.compile(
@@ -152,8 +157,27 @@ def probe_bilibili(room_id: str) -> bool:
    return data.get("live_status") == 1
 def probe_douyu(room_id: str) -> bool:
    """show_status: 1 live, 2 offline; videoLoop carousel counts as offline."""
    payload = http_utils.http_json(
        DOUYU_BETARD_URL.format(room_id=room_id), timeout=TIMEOUT
    )
    if not isinstance(payload, dict):
        raise ValueError("douyu betard returned non-object")
    room = payload.get("room")
    if not isinstance(room, dict):
        raise ValueError("douyu betard returned no room")
    if int(room.get("videoLoop") or 0) == 1:
        return False
    status = room.get("show_status")
    try:
        return int(status) == 1
    except (TypeError, ValueError) as e:
        raise ValueError(f"douyu unexpected show_status {status!r}") from e
 def room_ref_from_url(live_url: str) -> str | None:
-    """First path segment of the live room URL (douyin web_rid / bilibili room id)."""
+    """First path segment of the live room URL (douyin/bilibili/douyu room id)."""
    path = urllib.parse.urlparse(live_url.strip()).path.strip("/")
    if not path:
        return None
@@ -167,6 +191,8 @@ def live_platform_from_url(live_url: str, fallback: str = "") -> str:
        return "bilibili"
    if "douyin.com" in host:
        return "douyin"
    if "douyu.com" in host:
        return "douyu"
    return (fallback or "").strip().lower()
@@ -210,6 +236,9 @@ def probe_streamers(
            elif platform == "bilibili":
                is_live = probe_bilibili(ref)
                time.sleep(BILIBILI_SPACING)
            elif platform == "douyu":
                is_live = probe_douyu(ref)
                time.sleep(DOUYU_SPACING)
            else:
                print(f"skip {sid}: platform={platform!r} unsupported", flush=True)
                continue
@@ -1,16 +1,23 @@
-"""Fetch Douyin profile fields into data/streamers.json.
+"""Fetch Douyin / Douyu profile fields into data/streamers.json.
 Manual seed fields (id / platform / live_url / profile_url / heroes / tagline)
 are preserved. Profile enrichment (nickname, signature, counts, avatar) is
-best-effort via Douyin HTML RENDER_DATA + text fallback; failures keep the
+best-effort:
-previous values.
+
 - Douyin: HTML RENDER_DATA + text fallback
 - Douyu: ``v.douyu.com/author/<hash>`` (or ``author-video/<hash>``) page
  ``window.$DATA`` (fans / following / plays / avatar). Room-only rows
  resolve ``up_id`` from the live-room HTML then fetch the author page;
  if that fails, fall back to ``betard`` (nickname / avatar / bio, no fans).
 Failures keep the previous values.
 Preview only — do not merge into relations/heroes or recommend.
 Part of refresh_web ``daily`` (soft-fail: never aborts the tier).
 Usage:
  python fetch_streamers.py
-  python fetch_streamers.py --ids xiaowang
+  python fetch_streamers.py --ids xiaowang k9
  python fetch_streamers.py --out data/streamers.json
 """
@@ -32,8 +39,8 @@ import urllib.request
 from datetime import datetime, timezone
 from typing import Any
-from shared.http_utils import write_json_atomic
+from shared.http_utils import http_json, write_json_atomic
-from shared.paths import DATA, STREAMER_AVATARS
+from shared.paths import DATA, ROOT, STREAMER_AVATARS
 OUT = DATA / "streamers.json"
 AVATAR_DIR = STREAMER_AVATARS
@@ -58,6 +65,17 @@ HTML_AVATAR_RE = re.compile(
    re.IGNORECASE,
 )
 SEC_UID_RE = re.compile(r"/user/(MS4wLjABAAAA[A-Za-z0-9_-]+)")
 DOUYU_AUTHOR_HASH_RE = re.compile(
    r"(?:v\.)?douyu\.com/author(?:-video)?/([A-Za-z0-9]+)", re.IGNORECASE
 )
 DOUYU_ROOM_RE = re.compile(
    r"(?:www\.)?douyu\.com/(\d+)(?:/|$|\?)", re.IGNORECASE
 )
 DOUYU_DATA_RE = re.compile(r"window\.\$DATA=(\{.*?\}),\$", re.DOTALL)
 DOUYU_BARE_KEY_RE = re.compile(r"([{\s,])([A-Za-z_][A-Za-z0-9_]*)\s*:")
 # Room HTML embeds up_id in plain JSON and/or JSON-escaped script strings.
 DOUYU_UP_ID_RE = re.compile(r'\\?"up_id\\?"\s*:\s*\\?"([A-Za-z0-9]+)\\?"')
 DOUYU_BETARD_URL = "https://www.douyu.com/betard/{room_id}"
 # Fields fetch may overwrite; manual seed keys are never removed.
 PROFILE_KEYS = (
    "nickname",
@@ -104,7 +122,11 @@ def _opener() -> urllib.request.OpenerDirector:
 def _get(
-    opener: urllib.request.OpenerDirector, url: str, *, timeout: int = 30
+    opener: urllib.request.OpenerDirector,
    url: str,
    *,
    timeout: int = 30,
    referer: str = "https://www.douyin.com/",
 ) -> tuple[str, str]:
    """Return (final_url, html)."""
    req = urllib.request.Request(
@@ -113,7 +135,7 @@ def _get(
            "User-Agent": BROWSER_UA,
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
-            "Referer": "https://www.douyin.com/",
+            "Referer": referer,
        },
    )
    with opener.open(req, timeout=timeout) as resp:
@@ -306,7 +328,7 @@ def resolve_profile_url(
    return url, None
-def download_avatar(url: str, dest: Path) -> bool:
+def download_avatar(url: str, dest: Path, *, referer: str | None = None) -> bool:
    dest.parent.mkdir(parents=True, exist_ok=True)
    # Prefer a larger CDN variant when the URL embeds a size token.
    candidates = [url]
@@ -314,12 +336,22 @@ def download_avatar(url: str, dest: Path) -> bool:
        candidates.insert(0, url.replace("/100x100/", "/720x720/"))
    if "300x300" in url:
        candidates.insert(0, url.replace("300x300", "720x720"))
    if "_avatar_middle." in url:
        candidates.insert(0, url.replace("_avatar_middle.", "_avatar_big."))
    if "_middle.jpg" in url:
        candidates.insert(0, url.replace("_middle.jpg", "_big.jpg"))
    host = urllib.parse.urlparse(url).netloc.lower()
    if referer is None:
        if "douyu" in host:
            referer = "https://www.douyu.com/"
        else:
            referer = "https://www.douyin.com/"
    for candidate in candidates:
        req = urllib.request.Request(
            candidate,
            headers={
                "User-Agent": BROWSER_UA,
-                "Referer": "https://www.douyin.com/",
+                "Referer": referer,
            },
        )
        try:
@@ -336,6 +368,153 @@ def download_avatar(url: str, dest: Path) -> bool:
    return False
 def parse_douyu_dollar_data(blob: str) -> dict:
    """Parse Douyu ``window.$DATA={...}`` JS object (bare keys) into a dict."""
    quoted = DOUYU_BARE_KEY_RE.sub(r'\1"\2":', blob.strip())
    data = json.loads(quoted)
    if not isinstance(data, dict):
        raise ValueError("douyu $DATA is not an object")
    return data
 def extract_profile_from_douyu_data(data: dict) -> dict:
    """Map Douyu author ``$DATA`` fields onto streamer profile keys."""
    out: dict[str, Any] = {}
    nick = _first_str(data.get("name"), data.get("nickname"))
    if nick:
        out["nickname"] = nick
    room_id = _first_str(data.get("roomId"), data.get("room_id"))
    if room_id:
        out["unique_id"] = room_id
        out["live_url"] = f"https://www.douyu.com/{room_id}"
    up_id = _first_str(data.get("upId"), data.get("up_id"))
    if up_id:
        # Canonical homepage (author space); author-video is an alias.
        out["profile_url"] = f"https://v.douyu.com/author/{up_id}"
    # Author bio only — do not fall back to live room title (would wipe
    # hand-seeded / betard signatures on every refresh).
    contents = _first_str(data.get("contents"), data.get("ownerAuthContents"))
    if contents:
        out["signature"] = contents
    out["following_count"] = _first_int(data.get("upFollowNum"), data.get("up_follow_num"))
    out["follower_count"] = _first_int(data.get("subscribeNum"), data.get("subscribe_num"))
    # Video play count — shown as「播放」on Douyu cards.
    out["total_favorited"] = _first_int(data.get("playCount"), data.get("play_count"))
    avatar = _first_str(data.get("avatar"))
    if avatar:
        out["avatar_url"] = avatar.replace(r"\/", "/")
    return out
 def fetch_douyu_author_profile(
    opener: urllib.request.OpenerDirector, profile_url: str
 ) -> dict:
    """Fetch Douyu author / author-video page and parse ``window.$DATA``."""
    m = DOUYU_AUTHOR_HASH_RE.search(profile_url)
    if not m:
        raise ValueError(f"not a douyu author url: {profile_url!r}")
    hash_id = m.group(1)
    # Prefer /author/; fall back to /author-video/ if $DATA is missing.
    errors: list[str] = []
    for path in (f"author/{hash_id}", f"author-video/{hash_id}"):
        page_url = f"https://v.douyu.com/{path}"
        try:
            _, html = _get(opener, page_url, referer="https://v.douyu.com/")
        except (urllib.error.URLError, TimeoutError, OSError) as e:
            errors.append(f"{path}: {e}")
            continue
        data_m = DOUYU_DATA_RE.search(html)
        if not data_m:
            errors.append(f"{path}: no window.$DATA")
            continue
        profile = extract_profile_from_douyu_data(
            parse_douyu_dollar_data(data_m.group(1))
        )
        if not any(
            profile.get(k) is not None
            for k in ("nickname", "follower_count", "avatar_url", "unique_id")
        ):
            errors.append(f"{path}: parsed empty")
            continue
        return profile
    raise ValueError("; ".join(errors) or "douyu author page failed")
 def extract_douyu_up_id_from_room_html(html: str) -> str | None:
    """Pull author ``up_id`` hash embedded in the live-room page HTML."""
    m = DOUYU_UP_ID_RE.search(html or "")
    return m.group(1) if m else None
 def resolve_douyu_author_url_from_room(
    opener: urllib.request.OpenerDirector, room_id: str
 ) -> str:
    """Fetch ``www.douyu.com/<rid>`` and build the author homepage URL."""
    page_url = f"https://www.douyu.com/{room_id}"
    _, html = _get(opener, page_url, referer="https://www.douyu.com/")
    up_id = extract_douyu_up_id_from_room_html(html)
    if not up_id:
        raise ValueError(f"douyu room {room_id}: no up_id in HTML")
    return f"https://v.douyu.com/author/{up_id}"
 def fetch_douyu_room_profile(room_id: str) -> dict:
    """Fallback enrichment from live-room ``betard`` (no fan counts)."""
    payload = http_json(DOUYU_BETARD_URL.format(room_id=room_id), timeout=20)
    if not isinstance(payload, dict):
        raise ValueError("douyu betard returned non-object")
    room = payload.get("room")
    if not isinstance(room, dict):
        raise ValueError("douyu betard returned no room")
    out: dict[str, Any] = {
        "unique_id": str(room_id),
        "live_url": f"https://www.douyu.com/{room_id}",
    }
    nick = _first_str(room.get("nickname"), room.get("owner_name"))
    if nick:
        out["nickname"] = nick
    details = _first_str(room.get("show_details"), room.get("room_name"))
    if details:
        out["signature"] = details
    avatar = room.get("avatar")
    avatar_url = None
    if isinstance(avatar, dict):
        avatar_url = _first_str(avatar.get("big"), avatar.get("middle"), avatar.get("small"))
    elif isinstance(avatar, str):
        avatar_url = avatar
    if not avatar_url:
        avatar_url = _first_str(room.get("owner_avatar"), room.get("avatar_mid"))
    if avatar_url:
        out["avatar_url"] = avatar_url
    if not out.get("nickname") and not out.get("avatar_url"):
        raise ValueError("douyu betard parsed empty")
    return out
 def fetch_douyu_profile(
    opener: urllib.request.OpenerDirector, row: dict
 ) -> dict:
    """Enrich a Douyu streamer from author URL, room→up_id, else betard."""
    profile_url = str(row.get("profile_url") or "").strip()
    live_url = str(row.get("live_url") or "").strip()
    if DOUYU_AUTHOR_HASH_RE.search(profile_url):
        return fetch_douyu_author_profile(opener, profile_url)
    room_id = None
    for candidate in (profile_url, live_url):
        m = DOUYU_ROOM_RE.search(candidate)
        if m:
            room_id = m.group(1)
            break
    if not room_id:
        raise ValueError("douyu row needs author profile_url or room live_url")
    try:
        author_url = resolve_douyu_author_url_from_room(opener, room_id)
        return fetch_douyu_author_profile(opener, author_url)
    except (urllib.error.URLError, TimeoutError, OSError, ValueError) as e:
        print(f"  douyu room→author failed ({e}); betard fallback", flush=True)
        return fetch_douyu_room_profile(room_id)
 def fetch_douyin_profile(
    opener: urllib.request.OpenerDirector, profile_url: str
 ) -> dict:
@@ -382,6 +561,15 @@ def merge_profile(row: dict, profile: dict, *, streamer_id: str) -> None:
        if val is None or val == "":
            continue
        row[key] = val
    # Fill missing live/profile URLs from platform enrichment; never wipe seeds.
    for key in ("live_url", "profile_url"):
        val = profile.get(key)
        if isinstance(val, str) and val and not str(row.get(key) or "").strip():
            row[key] = val
    # Prefer canonical Douyu author homepage when enrichment found one.
    prof = profile.get("profile_url")
    if isinstance(prof, str) and DOUYU_AUTHOR_HASH_RE.search(prof):
        row["profile_url"] = prof
    avatar_url = profile.get("avatar_url")
    if isinstance(avatar_url, str) and avatar_url:
        dest = AVATAR_DIR / f"{streamer_id}.jpg"
@@ -408,18 +596,25 @@ def enrich_streamers(
        if ids is not None and sid not in ids:
            continue
        platform = str(row.get("platform") or "").strip().lower()
-        if platform != "douyin":
+        if platform not in ("douyin", "douyu"):
-            print(f"skip {sid}: platform={platform!r} (only douyin supported)", flush=True)
+            print(
                f"skip {sid}: platform={platform!r} "
                f"(supported: douyin, douyu)",
                flush=True,
            )
            skip += 1
            continue
-        profile_url = str(row.get("profile_url") or "").strip()
+        print(f"fetching {sid} ({platform}) ...", flush=True)
        if not profile_url:
            print(f"skip {sid}: missing profile_url", flush=True)
            skip += 1
            continue
        print(f"fetching {sid} ...", flush=True)
        try:
-            profile = fetch_douyin_profile(opener, profile_url)
+            if platform == "douyin":
                profile_url = str(row.get("profile_url") or "").strip()
                if not profile_url:
                    print(f"skip {sid}: missing profile_url", flush=True)
                    skip += 1
                    continue
                profile = fetch_douyin_profile(opener, profile_url)
            else:
                profile = fetch_douyu_profile(opener, row)
            merge_profile(row, profile, streamer_id=sid)
            nick = row.get("nickname") or "?"
            print(
@@ -434,7 +629,7 @@ def enrich_streamers(
        time.sleep(0.8)
    if ok > 0:
        payload["fetched_at"] = _now_iso()
-    payload["source"] = payload.get("source") or "manual+douyin"
+    payload["source"] = payload.get("source") or "manual+douyin+douyu"
    meta = payload.get("platform_meta")
    if not isinstance(meta, dict):
        meta = {}
@@ -442,13 +637,19 @@ def enrich_streamers(
        "douyin",
        {"label_zh": "抖音", "icon": "ui-icon/platform_douyin.png"},
    )
    meta.setdefault(
        "douyu",
        {"label_zh": "斗鱼", "icon": "ui-icon/platform_douyu.png"},
    )
    payload["platform_meta"] = meta
    _ = PROFILE_KEYS
    return ok, skip, fail
 def main() -> int:
-    ap = argparse.ArgumentParser(description="Enrich streamers.json from Douyin profiles")
+    ap = argparse.ArgumentParser(
        description="Enrich streamers.json from Douyin / Douyu profiles"
    )
    ap.add_argument("--out", type=Path, default=OUT)
    ap.add_argument(
        "--ids",
@@ -4364,7 +4364,14 @@ function buildStreamerMetaLine(row) {
  const parts = [];
  if (uniqueId) {
    parts.push({
-      label: platform === "douyin" ? "抖音号" : platform === "bilibili" ? "UID" : "账号",
+      label:
        platform === "douyin"
          ? "抖音号"
          : platform === "bilibili"
            ? "UID"
            : platform === "douyu"
              ? "房间号"
              : "账号",
      value: String(uniqueId),
      account: true,
    });
@@ -4373,6 +4380,13 @@ function buildStreamerMetaLine(row) {
    if (Number.isFinite(Number(row.following_count))) {
      parts.push({ label: "关注", value: formatStreamerCount(row.following_count) });
    }
  } else if (platform === "douyu") {
    if (Number.isFinite(Number(row.following_count))) {
      parts.push({ label: "关注", value: formatStreamerCount(row.following_count) });
    }
    if (Number.isFinite(Number(row.total_favorited))) {
      parts.push({ label: "播放", value: formatStreamerCount(row.total_favorited) });
    }
  } else if (Number.isFinite(Number(row.total_favorited))) {
    parts.push({ label: "获赞", value: formatStreamerCount(row.total_favorited) });
  }
@@ -1,5 +1,5 @@
 /* Local defaults; production export overwrites via export_relations_site.py. */
-var SITE_VERSION = "0.5.112";
+var SITE_VERSION = "0.5.114";
 var ABILITY_VIDEO_BASE = "";
 var STATIC_ASSET_BASE = "";
@@ -10,6 +10,8 @@
 * Probe logic is a JS port of fetch_streamer_live.py:
 * - Bilibili: api.live.bilibili.com Room/get_info; data.live_status === 1 is
 *   live (0 offline, 2 replay counts as offline).
 * - Douyu: www.douyu.com/betard/<room_id>; room.show_status === 1 is live
 *   (2 offline); room.videoLoop === 1 (carousel) counts as offline.
 * - Douyin: warm up cookies (www.douyin.com + live.douyin.com), then GET
 *   live.douyin.com/<web_rid> with a browser UA and parse the escaped JSON in
 *   the SSR pace chunks: roomStore.roomInfo.room.status (2 live / 4 offline);
@@ -44,6 +46,7 @@ const DOUYIN_HOME = "https://www.douyin.com/";
 const DOUYIN_LIVE_HOME = "https://live.douyin.com/";
 const BILIBILI_INFO_URL =
  "https://api.live.bilibili.com/room/v1/Room/get_info?room_id=";
 const DOUYU_BETARD_URL = "https://www.douyu.com/betard/";
 // Escaped JSON inside the SSR pace chunks: \"roomStore\":{\"roomInfo\":{\"room\":{
 const DOUYIN_ROOMSTORE_RE = /\\"roomStore\\":\s*\{\\"roomInfo\\":\s*\{\\"room\\":\s*\{/;
@@ -162,7 +165,25 @@ export async function probeBilibiliRoom(roomId) {
  return data.live_status === 1;
 }
-/** First path segment of the live room URL (douyin web_rid / bilibili room id). */
+/** show_status: 1 live, 2 offline; videoLoop carousel counts as offline. */
 export async function probeDouyuRoom(roomId) {
  const res = await fetchWithTimeout(DOUYU_BETARD_URL + roomId, {
    headers: {
      "User-Agent": BROWSER_UA,
      Accept: "application/json",
      Referer: "https://www.douyu.com/",
    },
  });
  const payload = await res.json();
  const room = payload && payload.room;
  if (!room || typeof room !== "object") {
    throw new Error("douyu betard returned no room");
  }
  if (Number(room.videoLoop || 0) === 1) return false;
  return Number(room.show_status) === 1;
 }
 /** First path segment of the live room URL (douyin/bilibili/douyu room id). */
 export function roomRefFromUrl(liveUrl) {
  let path = "";
  try {
@@ -180,6 +201,7 @@ export function livePlatformFromUrl(liveUrl, fallback = "") {
    const host = new URL(String(liveUrl).trim()).hostname.toLowerCase();
    if (host.includes("bilibili.com")) return "bilibili";
    if (host.includes("douyin.com")) return "douyin";
    if (host.includes("douyu.com")) return "douyu";
  } catch {
    /* keep fallback */
  }
@@ -245,6 +267,7 @@ async function probeAll(targets) {
          let isLive;
          if (t.platform === "douyin") isLive = await probeDouyinRoom(t.ref, jar);
          else if (t.platform === "bilibili") isLive = await probeBilibiliRoom(t.ref);
          else if (t.platform === "douyu") isLive = await probeDouyuRoom(t.ref);
          else throw new Error(`unsupported platform ${t.platform}`);
          results.set(t.id, { is_live: isLive });
        } catch (err) {
@@ -5,8 +5,8 @@
  <meta name="viewport" content="width=device-width, initial-scale=1" />
  <title>DOTA2 上分帝</title>
  <link rel="icon" href="/ui-icon/dota2_logo.png" type="image/png" />
-  <link rel="stylesheet" href="/style.css?v=0.5.112" />
+  <link rel="stylesheet" href="/style.css?v=0.5.114" />
-  <script src="/mobile-gate.js?v=0.5.112"></script>
+  <script src="/mobile-gate.js?v=0.5.114"></script>
 </head>
 <body>
  <h1 class="sr-only">DOTA2 上分帝</h1>
@@ -178,8 +178,8 @@
  <section class="detail" id="detail" aria-live="polite"></section>
-  <script src="/config.js?v=0.5.112"></script>
+  <script src="/config.js?v=0.5.114"></script>
-  <script src="/router.js?v=0.5.112"></script>
+  <script src="/router.js?v=0.5.114"></script>
-  <script src="/app.js?v=0.5.112"></script>
+  <script src="/app.js?v=0.5.114"></script>
 </body>
 </html>
@@ -17,6 +17,12 @@ sys.path.insert(0, str(ROOT / "web"))
 from web import notify_site_traffic, refresh_web
 from web.fetch_stratz_meta import preserve_failed_meta_brackets
 from web.fetch_streamer_live import probe_streamers
 from web.fetch_streamers import (
    DOUYU_AUTHOR_HASH_RE,
    extract_douyu_up_id_from_room_html,
    extract_profile_from_douyu_data,
    parse_douyu_dollar_data,
 )
 from web.refresh_cache import restore_patches, restore_streamers
 from shared.http_utils import write_json_atomic
 from shared.paths import HEROES_JSON, RELATIONS_JSON, WEB_FRONTEND
@@ -260,6 +266,65 @@ class LiveProbeFallbackTests(unittest.TestCase):
        self.assertFalse(row["is_live"])
        self.assertNotIn("live_probed_at", row)
    def test_douyu_probe_path_writes_live_flag(self) -> None:
        payload = {
            "streamers": [
                {
                    "id": "gouhuang",
                    "platform": "douyu",
                    "live_url": "https://www.douyu.com/957090",
                }
            ]
        }
        with patch("web.fetch_streamer_live.probe_douyu", return_value=True):
            live, offline, fail = probe_streamers(payload)
        self.assertEqual((live, offline, fail), (1, 0, 0))
        self.assertTrue(payload["streamers"][0]["is_live"])
        self.assertIn("live_probed_at", payload["streamers"][0])
 class DouyuProfileParseTests(unittest.TestCase):
    def test_author_and_author_video_hashes(self) -> None:
        self.assertEqual(
            DOUYU_AUTHOR_HASH_RE.search(
                "https://v.douyu.com/author/JPw9YOLKlw5X"
            ).group(1),
            "JPw9YOLKlw5X",
        )
        self.assertEqual(
            DOUYU_AUTHOR_HASH_RE.search(
                "https://v.douyu.com/author-video/JPw9YOLKlw5X"
            ).group(1),
            "JPw9YOLKlw5X",
        )
    def test_dollar_data_maps_fan_fields(self) -> None:
        blob = (
            '{type:"0",uid:"13506421",upId:"JPw9YOLKlw5X",upFollowNum:"16",'
            'subscribeNum:"465781",playCount:"26074077",'
            'avatar:"https://example.com/a.jpg",'
            'name: "18yearsold\\u5929\\u6b8b\\u5c11\\u5e74K9",'
            'contents: "",roomId:"235520",roomName: "k9 room"}'
        )
        profile = extract_profile_from_douyu_data(parse_douyu_dollar_data(blob))
        self.assertEqual(profile["nickname"], "18yearsold天残少年K9")
        self.assertEqual(profile["unique_id"], "235520")
        self.assertEqual(profile["following_count"], 16)
        self.assertEqual(profile["follower_count"], 465781)
        self.assertEqual(profile["total_favorited"], 26074077)
        self.assertEqual(profile["profile_url"], "https://v.douyu.com/author/JPw9YOLKlw5X")
        self.assertEqual(profile["live_url"], "https://www.douyu.com/235520")
        self.assertNotIn("signature", profile)
    def test_room_html_up_id_extraction(self) -> None:
        plain = '{"rid":957090,"up_id":"EqAvg1lQD75L","ban_display":0}'
        escaped = (
            r'{\"rid\":957090,\"up_id\":\"EqAvg1lQD75L\",\"ban_display\":0}'
        )
        self.assertEqual(extract_douyu_up_id_from_room_html(plain), "EqAvg1lQD75L")
        self.assertEqual(extract_douyu_up_id_from_room_html(escaped), "EqAvg1lQD75L")
        self.assertIsNone(extract_douyu_up_id_from_room_html("<html></html>"))
 class NotifySummaryTests(unittest.TestCase):
    def test_skipped_summary_is_labeled(self) -> None: