MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models
The rapid rise of Large Language Models (LLMs)-based intelligent agents
underscores the need for robust, scalable evaluation frameworks. Existing
methods rely on static benchmarks and labor-intensive data collection, limiting
practical assessment. We introduce \oursystemname, an open-source Model Context
Protocol (MCP)-based framework that automates end-to-end task generation and
deep evaluation of LLM agents across diverse domains. MCPEval standardizes
metrics, seamlessly integrates with native agent tools, and eliminates manual
effort in building evaluation pipelines. Empirical results across five
real-world domains show its effectiveness in revealing nuanced, domain-specific
performance. We publicly release MCPEval
https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and
standardized LLM agent evaluation.
687fd07d6b4cbf38ad40e0e0
Zhiwei Liu
687fd07d6b4cbf38ad40e0e1
Jielin Qiu
claimed_verified
2025-07-23T08:37:02.054Z
63294eeace50f125375243c1
/avatars/1a5e3b5552e9127d4e14d85880b122bd.svg
Jielin Qiu
user
jasonqiu
687fd07d6b4cbf38ad40e0e2
Shiyu Wang
687fd07d6b4cbf38ad40e0e3
Jianguo Zhang
687fd07d6b4cbf38ad40e0e4
Zuxin Liu
687fd07d6b4cbf38ad40e0e5
Roshan Ram
687fd07d6b4cbf38ad40e0e6
Haolin Chen
687fd07d6b4cbf38ad40e0e7
Weiran Yao
claimed_verified
2025-07-23T08:37:05.376Z
632cdea254e2c512c8f95b12
/avatars/a6d06cdd75861ae7d589f1343d81a5c5.svg
Weiran Yao
user
weirayao
687fd07d6b4cbf38ad40e0e8
Huan Wang
687fd07d6b4cbf38ad40e0e9
Shelby Heinecke
687fd07d6b4cbf38ad40e0ea
Silvio Savarese
687fd07d6b4cbf38ad40e0eb
Caiming Xiong
MCPEval: Automatic MCP-based Deep Evaluation for AI Agent Models
The rapid rise of Large Language Models (LLMs)-based intelligent agents
underscores the need for robust, scalable evaluation frameworks. Existing
methods rely on static benchmarks and labor-intensive data collection, limiting
practical assessment. We introduce \oursystemname, an open-source Model Context
Protocol (MCP)-based framework that automates end-to-end task generation and
deep evaluation of LLM agents across diverse domains. MCPEval standardizes
metrics, seamlessly integrates with native agent tools, and eliminates manual
effort in building evaluation pipelines. Empirical results across five
real-world domains show its effectiveness in revealing nuanced, domain-specific
performance. We publicly release MCPEval
https://github.com/SalesforceAIResearch/MCPEval to promote reproducible and
standardized LLM agent evaluation.
MCPEval is an open-source framework that automates task generation and evaluation for Large Language Models across various domains, improving the assessment process.
Large Language Models Model Context Protocol MCPEval task generation deep evaluation reproducible evaluation