Agent-as-a-Judge: Evaluate Agents with Agents
Contemporary evaluation techniques are inadequate for agentic systems. These
approaches either focus exclusively on final outcomes -- ignoring the
step-by-step nature of agentic systems, or require excessive manual labour. To
address this, we introduce the Agent-as-a-Judge framework, wherein agentic
systems are used to evaluate agentic systems. This is an organic extension of
the LLM-as-a-Judge framework, incorporating agentic features that enable
intermediate feedback for the entire task-solving process. We apply the
Agent-as-a-Judge to the task of code generation. To overcome issues with
existing benchmarks and provide a proof-of-concept testbed for
Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
AI development tasks. It includes rich manual annotations, like a total of 365
hierarchical user requirements. We benchmark three of the popular agentic
systems using Agent-as-a-Judge and find it dramatically outperforms
LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
we believe that Agent-as-a-Judge marks a concrete step forward for modern
agentic systems -- by providing rich and reliable reward signals necessary for
dynamic and scalable self-improvement.
670fc95b5a714a5429d3a214
Mingchen Zhuge
claimed_verified
2025-04-04T07:26:26.780Z
64403daae44f30a72323e4ca
https://cdn-avatars.huggingface.co/v1/production/uploads/64403daae44f30a72323e4ca/skJ9h0pdNfmE4VbQL8xDR.png
mingchen zhuge
user
tjpxiaoming
670fc95b5a714a5429d3a215
Changsheng Zhao
670fc95b5a714a5429d3a216
Dylan Ashley
670fc95b5a714a5429d3a217
Wenyi Wang
670fc95b5a714a5429d3a218
Dmitrii Khizbullin
670fc95b5a714a5429d3a219
Yunyang Xiong
670fc95b5a714a5429d3a21a
Zechun Liu
670fc95b5a714a5429d3a21b
Ernie Chang
670fc95b5a714a5429d3a21c
Raghuraman Krishnamoorthi
670fc95b5a714a5429d3a21d
Yuandong Tian
670fc95b5a714a5429d3a21e
Yangyang Shi
670fc95b5a714a5429d3a21f
Vikas Chandra
670fc95b5a714a5429d3a220
Jürgen Schmidhuber
Agent-as-a-Judge: Evaluate Agents with Agents
Contemporary evaluation techniques are inadequate for agentic systems. These
approaches either focus exclusively on final outcomes -- ignoring the
step-by-step nature of agentic systems, or require excessive manual labour. To
address this, we introduce the Agent-as-a-Judge framework, wherein agentic
systems are used to evaluate agentic systems. This is an organic extension of
the LLM-as-a-Judge framework, incorporating agentic features that enable
intermediate feedback for the entire task-solving process. We apply the
Agent-as-a-Judge to the task of code generation. To overcome issues with
existing benchmarks and provide a proof-of-concept testbed for
Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated
AI development tasks. It includes rich manual annotations, like a total of 365
hierarchical user requirements. We benchmark three of the popular agentic
systems using Agent-as-a-Judge and find it dramatically outperforms
LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether,
we believe that Agent-as-a-Judge marks a concrete step forward for modern
agentic systems -- by providing rich and reliable reward signals necessary for
dynamic and scalable self-improvement.
Agent-as-a-Judge framework uses agentic systems for evaluation, offering intermediate feedback and reliability improvements over existing methods like LLM-as-a-Judge in tasks such as code generation.
Agent-as-a-Judge LLM-as-a-Judge agentic systems code generation DevAI user requirements