title: Agent-as-a-Judge: Evaluate Agents with Agents
summary: Contemporary evaluation techniques are inadequate for agentic systems. These
  approaches either focus exclusively on final outcomes -- ignoring the
  step-by-step nature of agentic systems, or require excessive manual labour. To
  address this, we introduce the Agent-as-a-Judge framework, wherein agentic
  systems are used to evaluate agentic systems. This is an organic extension of
  the LLM-as-a-Judge framework, incorporating agentic features that enable
  intermediate feedback for the entire task-solving process. We apply the
  Agent-as-a-Judge to the task of code generation. To overcome issues with
  existing benchmarks and provide a proof-of-concept testbed for
  Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
  AI development tasks. It includes rich manual annotations, like a total of 365
  hierarchical user requirements. We benchmark three of the popular agentic
  systems using Agent-as-a-Judge and find it dramatically outperforms
  LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
  we believe that Agent-as-a-Judge marks a concrete step forward for modern
  agentic systems -- by providing rich and reliable reward signals necessary for
  dynamic and scalable self-improvement.
paper_authors._id: 670fc95b5a714a5429d3a214
paper_authors.name: Mingchen Zhuge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:26:26.780Z
paper_authors._id: 64403daae44f30a72323e4ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64403daae44f30a72323e4ca/skJ9h0pdNfmE4VbQL8xDR.png
paper_authors.fullname: mingchen zhuge
paper_authors.type: user
paper_authors.user: tjpxiaoming
paper_authors._id: 670fc95b5a714a5429d3a215
paper_authors.name: Changsheng Zhao
paper_authors._id: 670fc95b5a714a5429d3a216
paper_authors.name: Dylan Ashley
paper_authors._id: 670fc95b5a714a5429d3a217
paper_authors.name: Wenyi Wang
paper_authors._id: 670fc95b5a714a5429d3a218
paper_authors.name: Dmitrii Khizbullin
paper_authors._id: 670fc95b5a714a5429d3a219
paper_authors.name: Yunyang Xiong
paper_authors._id: 670fc95b5a714a5429d3a21a
paper_authors.name: Zechun Liu
paper_authors._id: 670fc95b5a714a5429d3a21b
paper_authors.name: Ernie Chang
paper_authors._id: 670fc95b5a714a5429d3a21c
paper_authors.name: Raghuraman Krishnamoorthi
paper_authors._id: 670fc95b5a714a5429d3a21d
paper_authors.name: Yuandong Tian
paper_authors._id: 670fc95b5a714a5429d3a21e
paper_authors.name: Yangyang Shi
paper_authors._id: 670fc95b5a714a5429d3a21f
paper_authors.name: Vikas Chandra
paper_authors._id: 670fc95b5a714a5429d3a220
paper_authors.name: Jürgen Schmidhuber
paper_title: Agent-as-a-Judge: Evaluate Agents with Agents
paper_summary: Contemporary evaluation techniques are inadequate for agentic systems. These
  approaches either focus exclusively on final outcomes -- ignoring the
  step-by-step nature of agentic systems, or require excessive manual labour. To
  address this, we introduce the Agent-as-a-Judge framework, wherein agentic
  systems are used to evaluate agentic systems. This is an organic extension of
  the LLM-as-a-Judge framework, incorporating agentic features that enable
  intermediate feedback for the entire task-solving process. We apply the
  Agent-as-a-Judge to the task of code generation. To overcome issues with
  existing benchmarks and provide a proof-of-concept testbed for
  Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
  AI development tasks. It includes rich manual annotations, like a total of 365
  hierarchical user requirements. We benchmark three of the popular agentic
  systems using Agent-as-a-Judge and find it dramatically outperforms
  LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
  we believe that Agent-as-a-Judge marks a concrete step forward for modern
  agentic systems -- by providing rich and reliable reward signals necessary for
  dynamic and scalable self-improvement.
paper_ai_summary: Agent-as-a-Judge framework uses agentic systems for evaluation, offering intermediate feedback and reliability improvements over existing methods like LLM-as-a-Judge in tasks such as code generation.
paper_ai_keywords: Agent-as-a-Judge LLM-as-a-Judge agentic systems code generation DevAI user requirements