LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?
With the rapid development of Model Context Protocol (MCP), the number of MCP
servers has surpassed 10,000. However, existing MCP benchmarks are limited to
single-server settings with only a few tools, hindering effective evaluation of
agent capabilities in large-scale, real-world scenarios. To address this
limitation, we present LiveMCPBench, the first comprehensive benchmark
comprising 95 real-world tasks grounded in the MCP ecosystem, designed to
evaluate LLM agents at scale across diverse servers. To support a scalable and
reproducible evaluation pipeline in large-scale MCP environments, we curate
LiveMCPTool, a diverse and readily deployable collection of 70 MCP servers and
527 tools. Furthermore, we introduce LiveMCPEval, an LLM-as-a-Judge framework
that enables automated and adaptive evaluation in dynamic, time-varying task
environments, achieving 81% agreement with human reviewers. Finally, we propose
the MCP Copilot Agent, a multi-step agent that routes tools for dynamic
planning and executes tools for API interaction across the entire LiveMCPTool
suite. Our evaluation covers 10 leading models, with the best-performing model
(Claude-Sonnet-4) reaching a 78.95% success rate. However, we observe large
performance variance across models, and several widely-used models perform
poorly in LiveMCPBench's complex, tool-rich environments. Overall, LiveMCPBench
offers the first unified framework for benchmarking LLM agents in realistic,
tool-rich, and dynamic MCP environments, laying a solid foundation for scalable
and reproducible research on agent capabilities. Our code and data will be
publicly available at https://icip-cas.github.io/LiveMCPBench.
689197dcf01a094725f83589
Guozhao Mo
689197dcf01a094725f8358a
Wenliang Zhong
689197dcf01a094725f8358b
Jiawei Chen
689197dcf01a094725f8358c
Xuanang Chen
689197dcf01a094725f8358d
Yaojie Lu
689197dcf01a094725f8358e
Hongyu Lin
689197dcf01a094725f8358f
Ben He
689197dcf01a094725f83590
Xianpei Han
689197dcf01a094725f83591
Le Sun
LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?
With the rapid development of Model Context Protocol (MCP), the number of MCP
servers has surpassed 10,000. However, existing MCP benchmarks are limited to
single-server settings with only a few tools, hindering effective evaluation of
agent capabilities in large-scale, real-world scenarios. To address this
limitation, we present LiveMCPBench, the first comprehensive benchmark
comprising 95 real-world tasks grounded in the MCP ecosystem, designed to
evaluate LLM agents at scale across diverse servers. To support a scalable and
reproducible evaluation pipeline in large-scale MCP environments, we curate
LiveMCPTool, a diverse and readily deployable collection of 70 MCP servers and
527 tools. Furthermore, we introduce LiveMCPEval, an LLM-as-a-Judge framework
that enables automated and adaptive evaluation in dynamic, time-varying task
environments, achieving 81% agreement with human reviewers. Finally, we propose
the MCP Copilot Agent, a multi-step agent that routes tools for dynamic
planning and executes tools for API interaction across the entire LiveMCPTool
suite. Our evaluation covers 10 leading models, with the best-performing model
(Claude-Sonnet-4) reaching a 78.95% success rate. However, we observe large
performance variance across models, and several widely-used models perform
poorly in LiveMCPBench's complex, tool-rich environments. Overall, LiveMCPBench
offers the first unified framework for benchmarking LLM agents in realistic,
tool-rich, and dynamic MCP environments, laying a solid foundation for scalable
and reproducible research on agent capabilities. Our code and data will be
publicly available at https://icip-cas.github.io/LiveMCPBench.
LiveMCPBench provides a comprehensive benchmark for evaluating LLM agents across a diverse set of real-world tasks in the MCP ecosystem, using a scalable evaluation pipeline and adaptive judging framework.
Model Context Protocol MCP LiveMCPBench LLM agents real-world tasks LiveMCPTool LiveMCPEval LLM-as-a-Judge MCP Copilot Agent API interaction dynamic task environments human reviewers performance variance