title: BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language
    Models in Chinese
summary: As large language models (LLMs) evolve into tool-using agents, the ability to
  browse the web in real-time has become a critical yardstick for measuring their
  reasoning and retrieval competence. Existing benchmarks such as BrowseComp
  concentrate on English and overlook the linguistic, infrastructural, and
  censorship-related complexities of other major information ecosystems -- most
  notably Chinese. To address this gap, we introduce BrowseComp-ZH, a
  high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents
  on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning
  11 diverse domains. Each question is reverse-engineered from a short,
  objective, and easily verifiable answer (e.g., a date, number, or proper noun).
  A two-stage quality control protocol is applied to strive for high question
  difficulty and answer uniqueness. We benchmark over 20 state-of-the-art
  language models and agentic search systems on our proposed BrowseComp-ZH.
  Despite their strong conversational and retrieval capabilities, most models
  struggle severely: a large number achieve accuracy rates below 10%, and only a
  handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch,
  reaches just 42.9%. These results demonstrate the considerable difficulty of
  BrowseComp-ZH, where success demands not only effective retrieval strategies,
  but also sophisticated reasoning and information reconciliation -- capabilities
  that current models still struggle to master. Our dataset, construction
  guidelines, and benchmark results have been publicly released at
  https://github.com/PALIN2018/BrowseComp-ZH.
paper_authors._id: 681d89e6d025518b321f67ce
paper_authors.name: Peilin Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:01:42.666Z
paper_authors._id: 6673cf668d570d59b83511cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/g9sJViXr1Cmx8A_is7Zgq.jpeg
paper_authors.fullname: Peilin Zhou
paper_authors.type: user
paper_authors.user: PALIN2018
paper_authors._id: 681d89e6d025518b321f67cf
paper_authors.name: Bruce Leon
paper_authors._id: 681d89e6d025518b321f67d0
paper_authors.name: Xiang Ying
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-05-09T04:51:51.507Z
paper_authors._id: 64489ca21d52a633c8f55aba
paper_authors.avatarUrl: /avatars/5199a5e93161c61d14ec13f79dd8c2c5.svg
paper_authors.fullname: Xiang Ying
paper_authors.type: user
paper_authors.user: MindYing
paper_authors._id: 681d89e6d025518b321f67d1
paper_authors.name: Can Zhang
paper_authors._id: 681d89e6d025518b321f67d2
paper_authors.name: Yifan Shao
paper_authors._id: 681d89e6d025518b321f67d3
paper_authors.name: Qichen Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:02:33.873Z
paper_authors._id: 636dfa6193d9a0c987d41b73
paper_authors.avatarUrl: /avatars/14396c8beb376b0d3c27a23fadaeb15e.svg
paper_authors.fullname: Qichen YE
paper_authors.type: user
paper_authors.user: yeeeqichen99
paper_authors._id: 681d89e6d025518b321f67d4
paper_authors.name: Dading Chong
paper_authors._id: 681d89e6d025518b321f67d5
paper_authors.name: Zhiling Jin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-09T09:02:48.180Z
paper_authors._id: 66bc683f432c73a183ef787c
paper_authors.avatarUrl: /avatars/9505a1e6131093a91d0454e50bcbba00.svg
paper_authors.fullname: Zhiling Jin
paper_authors.type: user
paper_authors.user: HawkFaust
paper_authors._id: 681d89e6d025518b321f67d6
paper_authors.name: Chenxuan Xie
paper_authors._id: 681d89e6d025518b321f67d7
paper_authors.name: Meng Cao
paper_authors._id: 681d89e6d025518b321f67d8
paper_authors.name: Yuxin Gu
paper_authors._id: 681d89e6d025518b321f67d9
paper_authors.name: Sixin Hong
paper_authors._id: 681d89e6d025518b321f67da
paper_authors.name: Jing Ren
paper_authors._id: 681d89e6d025518b321f67db
paper_authors.name: Jian Chen
paper_authors._id: 681d89e6d025518b321f67dc
paper_authors.name: Chao Liu
paper_authors._id: 681d89e6d025518b321f67dd
paper_authors.name: Yining Hua
paper_title: BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language
    Models in Chinese
paper_summary: As large language models (LLMs) evolve into tool-using agents, the ability to
  browse the web in real-time has become a critical yardstick for measuring their
  reasoning and retrieval competence. Existing benchmarks such as BrowseComp
  concentrate on English and overlook the linguistic, infrastructural, and
  censorship-related complexities of other major information ecosystems -- most
  notably Chinese. To address this gap, we introduce BrowseComp-ZH, a
  high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents
  on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning
  11 diverse domains. Each question is reverse-engineered from a short,
  objective, and easily verifiable answer (e.g., a date, number, or proper noun).
  A two-stage quality control protocol is applied to strive for high question
  difficulty and answer uniqueness. We benchmark over 20 state-of-the-art
  language models and agentic search systems on our proposed BrowseComp-ZH.
  Despite their strong conversational and retrieval capabilities, most models
  struggle severely: a large number achieve accuracy rates below 10%, and only a
  handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch,
  reaches just 42.9%. These results demonstrate the considerable difficulty of
  BrowseComp-ZH, where success demands not only effective retrieval strategies,
  but also sophisticated reasoning and information reconciliation -- capabilities
  that current models still struggle to master. Our dataset, construction
  guidelines, and benchmark results have been publicly released at
  https://github.com/PALIN2018/BrowseComp-ZH.
paper_ai_summary: BrowseComp-ZH evaluates large language models on real-time Chinese web browsing tasks, highlighting challenges in retrieval and reasoning beyond existing English benchmarks.
paper_ai_keywords: large language models tool-using agents BrowseComp BrowseComp-ZH multi-hop questions quality control protocol agentic search systems effective retrieval strategies sophisticated reasoning information reconciliation