BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language
  Models in Chinese
As large language models (LLMs) evolve into tool-using agents, the ability to
browse the web in real-time has become a critical yardstick for measuring their
reasoning and retrieval competence. Existing benchmarks such as BrowseComp
concentrate on English and overlook the linguistic, infrastructural, and
censorship-related complexities of other major information ecosystems -- most
notably Chinese. To address this gap, we introduce BrowseComp-ZH, a
high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents
on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning
11 diverse domains. Each question is reverse-engineered from a short,
objective, and easily verifiable answer (e.g., a date, number, or proper noun).
A two-stage quality control protocol is applied to strive for high question
difficulty and answer uniqueness. We benchmark over 20 state-of-the-art
language models and agentic search systems on our proposed BrowseComp-ZH.
Despite their strong conversational and retrieval capabilities, most models
struggle severely: a large number achieve accuracy rates below 10%, and only a
handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch,
reaches just 42.9%. These results demonstrate the considerable difficulty of
BrowseComp-ZH, where success demands not only effective retrieval strategies,
but also sophisticated reasoning and information reconciliation -- capabilities
that current models still struggle to master. Our dataset, construction
guidelines, and benchmark results have been publicly released at
https://github.com/PALIN2018/BrowseComp-ZH.
681d89e6d025518b321f67ce
Peilin Zhou
admin_assigned
2025-05-09T09:01:42.666Z
6673cf668d570d59b83511cc
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/g9sJViXr1Cmx8A_is7Zgq.jpeg
Peilin Zhou
user
PALIN2018
681d89e6d025518b321f67cf
Bruce Leon
681d89e6d025518b321f67d0
Xiang Ying
extracted_pending
2025-05-09T04:51:51.507Z
64489ca21d52a633c8f55aba
/avatars/5199a5e93161c61d14ec13f79dd8c2c5.svg
Xiang Ying
user
MindYing
681d89e6d025518b321f67d1
Can Zhang
681d89e6d025518b321f67d2
Yifan Shao
681d89e6d025518b321f67d3
Qichen Ye
admin_assigned
2025-05-09T09:02:33.873Z
636dfa6193d9a0c987d41b73
/avatars/14396c8beb376b0d3c27a23fadaeb15e.svg
Qichen YE
user
yeeeqichen99
681d89e6d025518b321f67d4
Dading Chong
681d89e6d025518b321f67d5
Zhiling Jin
admin_assigned
2025-05-09T09:02:48.180Z
66bc683f432c73a183ef787c
/avatars/9505a1e6131093a91d0454e50bcbba00.svg
Zhiling Jin
user
HawkFaust
681d89e6d025518b321f67d6
Chenxuan Xie
681d89e6d025518b321f67d7
Meng Cao
681d89e6d025518b321f67d8
Yuxin Gu
681d89e6d025518b321f67d9
Sixin Hong
681d89e6d025518b321f67da
Jing Ren
681d89e6d025518b321f67db
Jian Chen
681d89e6d025518b321f67dc
Chao Liu
681d89e6d025518b321f67dd
Yining Hua
BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language
  Models in Chinese
As large language models (LLMs) evolve into tool-using agents, the ability to
browse the web in real-time has become a critical yardstick for measuring their
reasoning and retrieval competence. Existing benchmarks such as BrowseComp
concentrate on English and overlook the linguistic, infrastructural, and
censorship-related complexities of other major information ecosystems -- most
notably Chinese. To address this gap, we introduce BrowseComp-ZH, a
high-difficulty benchmark purpose-built to comprehensively evaluate LLM agents
on the Chinese web. BrowseComp-ZH consists of 289 multi-hop questions spanning
11 diverse domains. Each question is reverse-engineered from a short,
objective, and easily verifiable answer (e.g., a date, number, or proper noun).
A two-stage quality control protocol is applied to strive for high question
difficulty and answer uniqueness. We benchmark over 20 state-of-the-art
language models and agentic search systems on our proposed BrowseComp-ZH.
Despite their strong conversational and retrieval capabilities, most models
struggle severely: a large number achieve accuracy rates below 10%, and only a
handful exceed 20%. Even the best-performing system, OpenAI's DeepResearch,
reaches just 42.9%. These results demonstrate the considerable difficulty of
BrowseComp-ZH, where success demands not only effective retrieval strategies,
but also sophisticated reasoning and information reconciliation -- capabilities
that current models still struggle to master. Our dataset, construction
guidelines, and benchmark results have been publicly released at
https://github.com/PALIN2018/BrowseComp-ZH.
BrowseComp-ZH evaluates large language models on real-time Chinese web browsing tasks, highlighting challenges in retrieval and reasoning beyond existing English benchmarks.
large language models tool-using agents BrowseComp BrowseComp-ZH multi-hop questions quality control protocol agentic search systems effective retrieval strategies sophisticated reasoning information reconciliation