Survey on Evaluation of LLM-based Agents
The emergence of LLM-based agents represents a paradigm shift in AI, enabling
autonomous systems to plan, reason, use tools, and maintain memory while
interacting with dynamic environments. This paper provides the first
comprehensive survey of evaluation methodologies for these increasingly capable
agents. We systematically analyze evaluation benchmarks and frameworks across
four critical dimensions: (1) fundamental agent capabilities, including
planning, tool use, self-reflection, and memory; (2) application-specific
benchmarks for web, software engineering, scientific, and conversational
agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating
agents. Our analysis reveals emerging trends, including a shift toward more
realistic, challenging evaluations with continuously updated benchmarks. We
also identify critical gaps that future research must address-particularly in
assessing cost-efficiency, safety, and robustness, and in developing
fine-grained, and scalable evaluation methods. This survey maps the rapidly
evolving landscape of agent evaluation, reveals the emerging trends in the
field, identifies current limitations, and proposes directions for future
research.
67dd1d595fd14aedd30bb94a
Asaf Yehudai
claimed_verified
2025-03-21T11:37:52.838Z
638324f862badff43269e588
/avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
Asaf Yehudai
user
Asaf-Yehudai
67dd1d595fd14aedd30bb94b
Lilach Eden
admin_assigned
2025-03-21T12:53:28.129Z
6694bdae1a0138cdbfc2799b
/avatars/8c42646280c327253493edc149d16a56.svg
Lilach Eden
user
LilachE
67dd1d595fd14aedd30bb94c
Alan Li
claimed_verified
2025-03-21T11:37:48.565Z
6209ab721c19916805e5b2d1
/avatars/67bbb958f19cedb2ea09461127f8ac92.svg
Alan Li
user
lihaoxin2020
67dd1d595fd14aedd30bb94d
Guy Uziel
67dd1d595fd14aedd30bb94e
Yilun Zhao
admin_assigned
2025-03-21T12:53:35.221Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
67dd1d595fd14aedd30bb94f
Roy Bar-Haim
admin_assigned
2025-03-21T12:53:41.335Z
666d8753e70e5838d9716446
/avatars/6390f9069372a72e79cd331bf043908b.svg
Roy Bar Haim
user
roybarhaim
67dd1d595fd14aedd30bb950
Arman Cohan
admin_assigned
2025-03-21T12:53:51.866Z
5f5ba21188f57f65f951f255
https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
Arman Cohan
user
armanc
67dd1d595fd14aedd30bb951
Michal Shmueli-Scheuer
Survey on Evaluation of LLM-based Agents
The emergence of LLM-based agents represents a paradigm shift in AI, enabling
autonomous systems to plan, reason, use tools, and maintain memory while
interacting with dynamic environments. This paper provides the first
comprehensive survey of evaluation methodologies for these increasingly capable
agents. We systematically analyze evaluation benchmarks and frameworks across
four critical dimensions: (1) fundamental agent capabilities, including
planning, tool use, self-reflection, and memory; (2) application-specific
benchmarks for web, software engineering, scientific, and conversational
agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating
agents. Our analysis reveals emerging trends, including a shift toward more
realistic, challenging evaluations with continuously updated benchmarks. We
also identify critical gaps that future research must address-particularly in
assessing cost-efficiency, safety, and robustness, and in developing
fine-grained, and scalable evaluation methods. This survey maps the rapidly
evolving landscape of agent evaluation, reveals the emerging trends in the
field, identifies current limitations, and proposes directions for future
research.
This survey analyzes evaluation methodologies for large language model-based agents, covering fundamental capabilities, application-specific benchmarks, and generalist agents, highlighting trends and gaps in the field.
LLM-based agents autonomous systems evaluation methodologies planning tool use self-reflection memory web agents software engineering agents scientific agents conversational agents generalist agents evaluation frameworks realistic evaluations continuously updated benchmarks cost-efficiency safety robustness fine-grained evaluation methods scalable evaluation methods