title: AgentRewardBench: Evaluating Automatic Evaluations of Web Agent
    Trajectories
summary: Web agents enable users to perform tasks on web browsers through natural
  language interaction. Evaluating web agents trajectories is an important
  problem, since it helps us determine whether the agent successfully completed
  the tasks. Rule-based methods are widely used for this purpose, but they are
  challenging to extend to new tasks and may not always recognize successful
  trajectories. We may achieve higher accuracy through human evaluation, but the
  process would be substantially slower and more expensive. Automatic evaluations
  with LLMs may avoid the challenges of designing new rules and manually
  annotating trajectories, enabling faster and cost-effective evaluation.
  However, it is unclear how effective they are at evaluating web agents. To this
  end, we propose AgentRewardBench, the first benchmark to assess the
  effectiveness of LLM judges for evaluating web agents. AgentRewardBench
  contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in
  AgentRewardBench is reviewed by an expert, who answers questions pertaining to
  the success, side effects, and repetitiveness of the agent. Using our
  benchmark, we evaluate 12 LLM judges and find that no single LLM excels across
  all benchmarks. We also find that the rule-based evaluation used by common
  benchmarks tends to underreport the success rate of web agents, highlighting a
  key weakness of rule-based evaluation and the need to develop more flexible
  automatic evaluations. We release the benchmark at:
  https://agent-reward-bench.github.io
paper_authors._id: 67fdadafdc27362617bbe714
paper_authors.name: Xing Han Lù
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:55:07.746Z
paper_authors._id: 5fa9ff3ea13e063b8b2b60cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1633380224986-5fa9ff3ea13e063b8b2b60cb.jpeg
paper_authors.fullname: Xing Han Lù
paper_authors.type: user
paper_authors.user: xhluca
paper_authors._id: 67fdadafdc27362617bbe715
paper_authors.name: Amirhossein Kazemnejad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:22:50.671Z
paper_authors._id: 63458f12d54fb141dedac508
paper_authors.avatarUrl: /avatars/3946fb9c23d1cd24037770cc0a3489bf.svg
paper_authors.fullname: Amirhossein Kazemnejad
paper_authors.type: user
paper_authors.user: kazemnejad
paper_authors._id: 67fdadafdc27362617bbe716
paper_authors.name: Nicholas Meade
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:22:56.742Z
paper_authors._id: 64527548fc4b47877aba7de0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64527548fc4b47877aba7de0/ht-mRRxNQT49A7NxArOGG.png
paper_authors.fullname: Nicholas Meade
paper_authors.type: user
paper_authors.user: ncmeade
paper_authors._id: 67fdadafdc27362617bbe717
paper_authors.name: Arkil Patel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:55:05.412Z
paper_authors._id: 631a523c04f8ed65eff16fb4
paper_authors.avatarUrl: /avatars/2b284403c88f140d7bef283f729f7a3e.svg
paper_authors.fullname: Arkil Patel
paper_authors.type: user
paper_authors.user: arkilpatel
paper_authors._id: 67fdadafdc27362617bbe718
paper_authors.name: Dongchan Shin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:23:03.255Z
paper_authors._id: 619af75e7812aec847ee7729
paper_authors.avatarUrl: /avatars/f50c05ee8b3105d20a8b291cc9f06ae4.svg
paper_authors.fullname: Dong Chan Shin
paper_authors.type: user
paper_authors.user: dongchans
paper_authors._id: 67fdadafdc27362617bbe719
paper_authors.name: Alejandra Zambrano
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:23:09.097Z
paper_authors._id: 65f5133599c842dd93b7bacd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/nxxfWGP_K0hf7BG1oM7c0.png
paper_authors.fullname: Alejandra Zambrano
paper_authors.type: user
paper_authors.user: alzambranolu
paper_authors._id: 67fdadafdc27362617bbe71a
paper_authors.name: Karolina Stańczak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:23:14.768Z
paper_authors._id: 60a66731e1db8bc33b8d4112
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60a66731e1db8bc33b8d4112/AY5Y0CnHh08u6lfEoQ6se.jpeg
paper_authors.fullname: Karolina Stanczak
paper_authors.type: user
paper_authors.user: Karolina
paper_authors._id: 67fdadafdc27362617bbe71b
paper_authors.name: Peter Shaw
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:23:22.550Z
paper_authors._id: 631cf223fe95faea33561d5f
paper_authors.avatarUrl: /avatars/ac0431955c6c5f4948461772a984a2ba.svg
paper_authors.fullname: Peter Shaw
paper_authors.type: user
paper_authors.user: PeterShaw
paper_authors._id: 67fdadafdc27362617bbe71c
paper_authors.name: Christopher J. Pal
paper_authors._id: 67fdadafdc27362617bbe71d
paper_authors.name: Siva Reddy
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-15T00:51:59.987Z
paper_authors._id: 624734dc4c731bb6bfab8af7
paper_authors.avatarUrl: /avatars/6b250b58710a3287b85e4733c1824558.svg
paper_authors.fullname: Siva Reddy
paper_authors.type: user
paper_authors.user: sivareddyg
paper_title: AgentRewardBench: Evaluating Automatic Evaluations of Web Agent
    Trajectories
paper_summary: Web agents enable users to perform tasks on web browsers through natural
  language interaction. Evaluating web agents trajectories is an important
  problem, since it helps us determine whether the agent successfully completed
  the tasks. Rule-based methods are widely used for this purpose, but they are
  challenging to extend to new tasks and may not always recognize successful
  trajectories. We may achieve higher accuracy through human evaluation, but the
  process would be substantially slower and more expensive. Automatic evaluations
  with LLMs may avoid the challenges of designing new rules and manually
  annotating trajectories, enabling faster and cost-effective evaluation.
  However, it is unclear how effective they are at evaluating web agents. To this
  end, we propose AgentRewardBench, the first benchmark to assess the
  effectiveness of LLM judges for evaluating web agents. AgentRewardBench
  contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in
  AgentRewardBench is reviewed by an expert, who answers questions pertaining to
  the success, side effects, and repetitiveness of the agent. Using our
  benchmark, we evaluate 12 LLM judges and find that no single LLM excels across
  all benchmarks. We also find that the rule-based evaluation used by common
  benchmarks tends to underreport the success rate of web agents, highlighting a
  key weakness of rule-based evaluation and the need to develop more flexible
  automatic evaluations. We release the benchmark at:
  https://agent-reward-bench.github.io
paper_ai_summary: AgentRewardBench is a benchmark evaluating the effectiveness of LLMs in assessing web agent performance, showing that rule-based methods may underreport success.
paper_ai_keywords: LLMs AgentRewardBench machine learning models natural language interaction web agents trajectory evaluation rule-based methods expert review