title: Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
summary: Web navigation is a unique domain that can automate many repetitive real-life
  tasks and is challenging as it requires long-horizon sequential decision making
  beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
  reward models for web navigation that can be utilized during both training and
  test-time have been absent until now. Despite the importance of speed and
  cost-effectiveness, prior works have utilized MLLMs as reward models, which
  poses significant constraints for real-world deployment. To address this, in
  this work, we propose the first process reward model (PRM) called Web-Shepherd
  which could assess web navigation trajectories in a step-level. To achieve
  this, we first construct the WebPRM Collection, a large-scale dataset with 40K
  step-level preference pairs and annotated checklists spanning diverse domains
  and difficulty levels. Next, we also introduce the WebRewardBench, the first
  meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
  that our Web-Shepherd achieves about 30 points better accuracy compared to
  using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
  using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
  10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
  as the verifier. Our model, dataset, and code are publicly available at LINK.
paper_authors._id: 682e854551706f69070aca6b
paper_authors.name: Hyungjoo Chae
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:37.301Z
paper_authors._id: 64c8f4cec547ed5243ebd0a8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c8f4cec547ed5243ebd0a8/MiOH5YbMg8Gh9KYlQsLmX.jpeg
paper_authors.fullname: Hyungjoo Chae
paper_authors.type: user
paper_authors.user: hyungjoochae
paper_authors._id: 682e854551706f69070aca6c
paper_authors.name: Sunghwan Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:32.322Z
paper_authors._id: 646a0897c37ca1e12308b026
paper_authors.avatarUrl: /avatars/6d720a9e366db9bec15c8c10878c0c75.svg
paper_authors.fullname: Sunghwan Kim
paper_authors.type: user
paper_authors.user: KimSHine
paper_authors._id: 682e854551706f69070aca6d
paper_authors.name: Junhee Cho
paper_authors._id: 682e854551706f69070aca6e
paper_authors.name: Seungone Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:55:27.225Z
paper_authors._id: 6469949654873f0043b09c22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
paper_authors.fullname: Seungone Kim
paper_authors.type: user
paper_authors.user: seungone
paper_authors._id: 682e854551706f69070aca6f
paper_authors.name: Seungjun Moon
paper_authors._id: 682e854551706f69070aca70
paper_authors.name: Gyeom Hwangbo
paper_authors._id: 682e854551706f69070aca71
paper_authors.name: Dongha Lim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-22T07:16:34.741Z
paper_authors._id: 6683b8680b72be136701de35
paper_authors.avatarUrl: /avatars/0c135e570b16b81ee2fb81ad65b01ba8.svg
paper_authors.fullname: Dongha Lim
paper_authors.type: user
paper_authors.user: donghalim
paper_authors._id: 682e854551706f69070aca72
paper_authors.name: Minjin Kim
paper_authors._id: 682e854551706f69070aca73
paper_authors.name: Yeonjun Hwang
paper_authors._id: 682e854551706f69070aca74
paper_authors.name: Minju Gwak
paper_authors._id: 682e854551706f69070aca75
paper_authors.name: Dongwook Choi
paper_authors._id: 682e854551706f69070aca76
paper_authors.name: Minseok Kang
paper_authors._id: 682e854551706f69070aca77
paper_authors.name: Gwanhoon Im
paper_authors._id: 682e854551706f69070aca78
paper_authors.name: ByeongUng Cho
paper_authors._id: 682e854551706f69070aca79
paper_authors.name: Hyojun Kim
paper_authors._id: 682e854551706f69070aca7a
paper_authors.name: Jun Hee Han
paper_authors._id: 682e854551706f69070aca7b
paper_authors.name: Taeyoon Kwon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:55:29.445Z
paper_authors._id: 636b529ef796304dd67d139c
paper_authors.avatarUrl: /avatars/7a64d5095fcb1da558b52ad48177ad76.svg
paper_authors.fullname: Taeyoon Kwon
paper_authors.type: user
paper_authors.user: Connoriginal
paper_authors._id: 682e854551706f69070aca7c
paper_authors.name: Minju Kim
paper_authors._id: 682e854551706f69070aca7d
paper_authors.name: Beong-woo Kwak
paper_authors._id: 682e854551706f69070aca7e
paper_authors.name: Dongjin Kang
paper_authors._id: 682e854551706f69070aca7f
paper_authors.name: Jinyoung Yeo
paper_title: Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
paper_summary: Web navigation is a unique domain that can automate many repetitive real-life
  tasks and is challenging as it requires long-horizon sequential decision making
  beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
  reward models for web navigation that can be utilized during both training and
  test-time have been absent until now. Despite the importance of speed and
  cost-effectiveness, prior works have utilized MLLMs as reward models, which
  poses significant constraints for real-world deployment. To address this, in
  this work, we propose the first process reward model (PRM) called Web-Shepherd
  which could assess web navigation trajectories in a step-level. To achieve
  this, we first construct the WebPRM Collection, a large-scale dataset with 40K
  step-level preference pairs and annotated checklists spanning diverse domains
  and difficulty levels. Next, we also introduce the WebRewardBench, the first
  meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
  that our Web-Shepherd achieves about 30 points better accuracy compared to
  using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
  using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
  10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
  as the verifier. Our model, dataset, and code are publicly available at LINK.
paper_ai_summary: The paper introduces Web-Shepherd, a process reward model for web navigation, which improves accuracy and cost-effectiveness in step-level trajectory assessment compared to existing multimodal large language models.
paper_ai_keywords: multimodal large language model process reward model web navigation webPRM collection webrewardbench long-horizon sequential decision making preference pairs annotated checklists step-level assessment webarena-lite policy verifier