title: Survey on Evaluation of LLM-based Agents
summary: The emergence of LLM-based agents represents a paradigm shift in AI, enabling
  autonomous systems to plan, reason, use tools, and maintain memory while
  interacting with dynamic environments. This paper provides the first
  comprehensive survey of evaluation methodologies for these increasingly capable
  agents. We systematically analyze evaluation benchmarks and frameworks across
  four critical dimensions: (1) fundamental agent capabilities, including
  planning, tool use, self-reflection, and memory; (2) application-specific
  benchmarks for web, software engineering, scientific, and conversational
  agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating
  agents. Our analysis reveals emerging trends, including a shift toward more
  realistic, challenging evaluations with continuously updated benchmarks. We
  also identify critical gaps that future research must address-particularly in
  assessing cost-efficiency, safety, and robustness, and in developing
  fine-grained, and scalable evaluation methods. This survey maps the rapidly
  evolving landscape of agent evaluation, reveals the emerging trends in the
  field, identifies current limitations, and proposes directions for future
  research.
paper_authors._id: 67dd1d595fd14aedd30bb94a
paper_authors.name: Asaf Yehudai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:37:52.838Z
paper_authors._id: 638324f862badff43269e588
paper_authors.avatarUrl: /avatars/907a39a9b44fc8b7f3fad35858b01fb7.svg
paper_authors.fullname: Asaf Yehudai
paper_authors.type: user
paper_authors.user: Asaf-Yehudai
paper_authors._id: 67dd1d595fd14aedd30bb94b
paper_authors.name: Lilach Eden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:53:28.129Z
paper_authors._id: 6694bdae1a0138cdbfc2799b
paper_authors.avatarUrl: /avatars/8c42646280c327253493edc149d16a56.svg
paper_authors.fullname: Lilach Eden
paper_authors.type: user
paper_authors.user: LilachE
paper_authors._id: 67dd1d595fd14aedd30bb94c
paper_authors.name: Alan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-21T11:37:48.565Z
paper_authors._id: 6209ab721c19916805e5b2d1
paper_authors.avatarUrl: /avatars/67bbb958f19cedb2ea09461127f8ac92.svg
paper_authors.fullname: Alan Li
paper_authors.type: user
paper_authors.user: lihaoxin2020
paper_authors._id: 67dd1d595fd14aedd30bb94d
paper_authors.name: Guy Uziel
paper_authors._id: 67dd1d595fd14aedd30bb94e
paper_authors.name: Yilun Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:53:35.221Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 67dd1d595fd14aedd30bb94f
paper_authors.name: Roy Bar-Haim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:53:41.335Z
paper_authors._id: 666d8753e70e5838d9716446
paper_authors.avatarUrl: /avatars/6390f9069372a72e79cd331bf043908b.svg
paper_authors.fullname: Roy Bar Haim
paper_authors.type: user
paper_authors.user: roybarhaim
paper_authors._id: 67dd1d595fd14aedd30bb950
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-21T12:53:51.866Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_authors._id: 67dd1d595fd14aedd30bb951
paper_authors.name: Michal Shmueli-Scheuer
paper_title: Survey on Evaluation of LLM-based Agents
paper_summary: The emergence of LLM-based agents represents a paradigm shift in AI, enabling
  autonomous systems to plan, reason, use tools, and maintain memory while
  interacting with dynamic environments. This paper provides the first
  comprehensive survey of evaluation methodologies for these increasingly capable
  agents. We systematically analyze evaluation benchmarks and frameworks across
  four critical dimensions: (1) fundamental agent capabilities, including
  planning, tool use, self-reflection, and memory; (2) application-specific
  benchmarks for web, software engineering, scientific, and conversational
  agents; (3) benchmarks for generalist agents; and (4) frameworks for evaluating
  agents. Our analysis reveals emerging trends, including a shift toward more
  realistic, challenging evaluations with continuously updated benchmarks. We
  also identify critical gaps that future research must address-particularly in
  assessing cost-efficiency, safety, and robustness, and in developing
  fine-grained, and scalable evaluation methods. This survey maps the rapidly
  evolving landscape of agent evaluation, reveals the emerging trends in the
  field, identifies current limitations, and proposes directions for future
  research.
paper_ai_summary: This survey analyzes evaluation methodologies for large language model-based agents, covering fundamental capabilities, application-specific benchmarks, and generalist agents, highlighting trends and gaps in the field.
paper_ai_keywords: LLM-based agents autonomous systems evaluation methodologies planning tool use self-reflection memory web agents software engineering agents scientific agents conversational agents generalist agents evaluation frameworks realistic evaluations continuously updated benchmarks cost-efficiency safety robustness fine-grained evaluation methods scalable evaluation methods