Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Agentic search such as Deep Research systems, where large language models
autonomously browse the web, synthesize information, and return comprehensive
citation-backed answers, represents a major shift in how users interact with
web-scale information. While promising greater efficiency and cognitive
offloading, the growing complexity and open-endedness of agentic search have
outpaced existing evaluation benchmarks and methodologies, which largely assume
short search horizons and static answers. In this paper, we introduce Mind2Web
2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that
require real-time web browsing and extensive information synthesis, constructed
with over 1,000 hours of human labor. To address the challenge of evaluating
time-varying and complex answers, we propose a novel Agent-as-a-Judge
framework. Our method constructs task-specific judge agents based on a
tree-structured rubric design to automatically assess both answer correctness
and source attribution. We conduct a comprehensive evaluation of nine frontier
agentic search systems and human performance, along with a detailed error
analysis to draw insights for future development. The best-performing system,
OpenAI Deep Research, can already achieve 50-70% of human performance while
spending half the time, showing a great potential. Altogether, Mind2Web 2
provides a rigorous foundation for developing and benchmarking the next
generation of agentic search systems.
685df93d71131fa43be08a96
Boyu Gou
claimed_verified
2025-06-27T08:57:06.439Z
6500870f1e14749e84f8f887
https://cdn-avatars.huggingface.co/v1/production/uploads/6500870f1e14749e84f8f887/wfvx4BZvh2OyW-vpq5jEy.jpeg
Boyu Gou
user
BoyuNLP
685df93d71131fa43be08a97
Zanming Huang
claimed_verified
2025-06-27T20:31:09.247Z
6745089cc681f914069f42a1
https://cdn-avatars.huggingface.co/v1/production/uploads/6745089cc681f914069f42a1/az6adRBIs8grHd0koJV1A.jpeg
Zanming Huang
user
huangtom
685df93d71131fa43be08a98
Yuting Ning
claimed_verified
2025-06-27T08:57:04.054Z
65ace92f64c9b93eca5c2bce
https://cdn-avatars.huggingface.co/v1/production/uploads/65ace92f64c9b93eca5c2bce/pG0JRXH-8zEy0IoaEnMNw.jpeg
Yuting Ning
user
nnnyt
685df93d71131fa43be08a99
Yu Gu
685df93d71131fa43be08a9a
Michael Lin
685df93d71131fa43be08a9b
Weijian Qi
685df93d71131fa43be08a9c
Andrei Kopanev
685df93d71131fa43be08a9d
Botao Yu
685df93d71131fa43be08a9e
Bernal Jiménez Gutiérrez
685df93d71131fa43be08a9f
Yiheng Shu
claimed_verified
2025-06-27T08:57:01.825Z
60a4ebfbaa9320dbbe69e37c
https://cdn-avatars.huggingface.co/v1/production/uploads/60a4ebfbaa9320dbbe69e37c/QLaEohXCWaUy8YX3wKQ_w.jpeg
Yiheng Shu
user
yhshu
685df93d71131fa43be08aa0
Chan Hee Song
claimed_verified
2025-06-27T20:31:06.896Z
63d19365b30415240fd6515b
https://cdn-avatars.huggingface.co/v1/production/uploads/63d19365b30415240fd6515b/eOEYSsyDTfPTDrR6Cm5Jn.jpeg
Chan Hee Song
user
chanhee-luke
685df93d71131fa43be08aa1
Jiaman Wu
685df93d71131fa43be08aa2
Shijie Chen
685df93d71131fa43be08aa3
Hanane Nour Moussa
claimed_verified
2025-06-27T20:31:03.818Z
62de6cea86220b5cb895c7ee
/avatars/f247687a5da264d3702c234bb4aee8ba.svg
Hanane Nour Moussa
user
hananour
685df93d71131fa43be08aa4
Tianshu Zhang
685df93d71131fa43be08aa5
Jian Xie
685df93d71131fa43be08aa6
Yifei Li
685df93d71131fa43be08aa7
Tianci Xue
685df93d71131fa43be08aa8
Zeyi Liao
685df93d71131fa43be08aa9
Kai Zhang
685df93d71131fa43be08aaa
Boyuan Zheng
685df93d71131fa43be08aab
Zhaowei Cai
685df93d71131fa43be08aac
Viktor Rozgic
685df93d71131fa43be08aad
Morteza Ziyadi
685df93d71131fa43be08aae
Huan Sun
685df93d71131fa43be08aaf
Yu Su
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
Agentic search such as Deep Research systems, where large language models
autonomously browse the web, synthesize information, and return comprehensive
citation-backed answers, represents a major shift in how users interact with
web-scale information. While promising greater efficiency and cognitive
offloading, the growing complexity and open-endedness of agentic search have
outpaced existing evaluation benchmarks and methodologies, which largely assume
short search horizons and static answers. In this paper, we introduce Mind2Web
2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that
require real-time web browsing and extensive information synthesis, constructed
with over 1,000 hours of human labor. To address the challenge of evaluating
time-varying and complex answers, we propose a novel Agent-as-a-Judge
framework. Our method constructs task-specific judge agents based on a
tree-structured rubric design to automatically assess both answer correctness
and source attribution. We conduct a comprehensive evaluation of nine frontier
agentic search systems and human performance, along with a detailed error
analysis to draw insights for future development. The best-performing system,
OpenAI Deep Research, can already achieve 50-70% of human performance while
spending half the time, showing a great potential. Altogether, Mind2Web 2
provides a rigorous foundation for developing and benchmarking the next
generation of agentic search systems.
Mind2Web 2 benchmark evaluates agentic search systems with a suite of realistic, long-horizon tasks, introducing an Agent-as-a-Judge framework to assess accuracy and source attribution.
Deep Research systems large language models autonomous browsing information synthesis citation-backed answers evaluation benchmarks search horizons static answers Mind2Web 2 high-quality tasks real-time web browsing extensive information synthesis task-specific judge agents tree-structured rubric design answer correctness source attribution agentic search systems human performance error analysis OpenAI Deep Research