Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Web navigation is a unique domain that can automate many repetitive real-life
tasks and is challenging as it requires long-horizon sequential decision making
beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
reward models for web navigation that can be utilized during both training and
test-time have been absent until now. Despite the importance of speed and
cost-effectiveness, prior works have utilized MLLMs as reward models, which
poses significant constraints for real-world deployment. To address this, in
this work, we propose the first process reward model (PRM) called Web-Shepherd
which could assess web navigation trajectories in a step-level. To achieve
this, we first construct the WebPRM Collection, a large-scale dataset with 40K
step-level preference pairs and annotated checklists spanning diverse domains
and difficulty levels. Next, we also introduce the WebRewardBench, the first
meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
that our Web-Shepherd achieves about 30 points better accuracy compared to
using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
as the verifier. Our model, dataset, and code are publicly available at LINK.
682e854551706f69070aca6b
Hyungjoo Chae
claimed_verified
2025-05-22T07:16:37.301Z
64c8f4cec547ed5243ebd0a8
https://cdn-avatars.huggingface.co/v1/production/uploads/64c8f4cec547ed5243ebd0a8/MiOH5YbMg8Gh9KYlQsLmX.jpeg
Hyungjoo Chae
user
hyungjoochae
682e854551706f69070aca6c
Sunghwan Kim
claimed_verified
2025-05-22T07:16:32.322Z
646a0897c37ca1e12308b026
/avatars/6d720a9e366db9bec15c8c10878c0c75.svg
Sunghwan Kim
user
KimSHine
682e854551706f69070aca6d
Junhee Cho
682e854551706f69070aca6e
Seungone Kim
claimed_verified
2025-05-27T07:55:27.225Z
6469949654873f0043b09c22
https://cdn-avatars.huggingface.co/v1/production/uploads/6469949654873f0043b09c22/Lk7IJAR16Wa_sGJ2g81AQ.jpeg
Seungone Kim
user
seungone
682e854551706f69070aca6f
Seungjun Moon
682e854551706f69070aca70
Gyeom Hwangbo
682e854551706f69070aca71
Dongha Lim
claimed_verified
2025-05-22T07:16:34.741Z
6683b8680b72be136701de35
/avatars/0c135e570b16b81ee2fb81ad65b01ba8.svg
Dongha Lim
user
donghalim
682e854551706f69070aca72
Minjin Kim
682e854551706f69070aca73
Yeonjun Hwang
682e854551706f69070aca74
Minju Gwak
682e854551706f69070aca75
Dongwook Choi
claimed_verified
2025-06-16T13:54:39.381Z
654c263fbe11400417c93d9f
/avatars/eb5778e28091200efee2c6b68589a1a2.svg
choi dongwook
user
dongwookchoi
682e854551706f69070aca76
Minseok Kang
682e854551706f69070aca77
Gwanhoon Im
682e854551706f69070aca78
ByeongUng Cho
682e854551706f69070aca79
Hyojun Kim
682e854551706f69070aca7a
Jun Hee Han
682e854551706f69070aca7b
Taeyoon Kwon
claimed_verified
2025-05-27T07:55:29.445Z
636b529ef796304dd67d139c
/avatars/7a64d5095fcb1da558b52ad48177ad76.svg
Taeyoon Kwon
user
Connoriginal
682e854551706f69070aca7c
Minju Kim
682e854551706f69070aca7d
Beong-woo Kwak
682e854551706f69070aca7e
Dongjin Kang
682e854551706f69070aca7f
Jinyoung Yeo
Web-Shepherd: Advancing PRMs for Reinforcing Web Agents
Web navigation is a unique domain that can automate many repetitive real-life
tasks and is challenging as it requires long-horizon sequential decision making
beyond typical multimodal large language model (MLLM) tasks. Yet, specialized
reward models for web navigation that can be utilized during both training and
test-time have been absent until now. Despite the importance of speed and
cost-effectiveness, prior works have utilized MLLMs as reward models, which
poses significant constraints for real-world deployment. To address this, in
this work, we propose the first process reward model (PRM) called Web-Shepherd
which could assess web navigation trajectories in a step-level. To achieve
this, we first construct the WebPRM Collection, a large-scale dataset with 40K
step-level preference pairs and annotated checklists spanning diverse domains
and difficulty levels. Next, we also introduce the WebRewardBench, the first
meta-evaluation benchmark for evaluating PRMs. In our experiments, we observe
that our Web-Shepherd achieves about 30 points better accuracy compared to
using GPT-4o on WebRewardBench. Furthermore, when testing on WebArena-lite by
using GPT-4o-mini as the policy and Web-Shepherd as the verifier, we achieve
10.9 points better performance, in 10 less cost compared to using GPT-4o-mini
as the verifier. Our model, dataset, and code are publicly available at LINK.
The paper introduces Web-Shepherd, a process reward model for web navigation, which improves accuracy and cost-effectiveness in step-level trajectory assessment compared to existing multimodal large language models.
multimodal large language model process reward model web navigation webPRM collection webrewardbench long-horizon sequential decision making preference pairs annotated checklists step-level assessment webarena-lite policy verifier