WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement
  Learning
While reinforcement learning (RL) has demonstrated remarkable success in
enhancing large language models (LLMs), it has primarily focused on single-turn
tasks such as solving math problems. Training effective web agents for
multi-turn interactions remains challenging due to the complexity of
long-horizon decision-making across dynamic web interfaces. In this work, we
present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework
for training web agents. It learns directly from online interactions with web
environments by asynchronously generating diverse trajectories, entirely guided
by binary rewards depending on task success. Experiments on the WebArena-Lite
benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task
success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to
44.8%, significantly outperforming existing state-of-the-art methods and strong
proprietary models such as OpenAI o3. In-depth analyses reveal the
effectiveness of the thinking-based prompting strategy and test-time scaling
through increased interactions for web tasks. We further investigate different
RL initialization policies by introducing two variants, namely WebAgent-R1-Zero
and WebAgent-R1-CoT, which highlight the importance of the warm-up training
stage (i.e., behavior cloning) and provide insights on incorporating long
chain-of-thought (CoT) reasoning in web agents.
682fe1e87d25524002b8a4a1
Zhepei Wei
682fe1e87d25524002b8a4a2
Wenlin Yao
682fe1e87d25524002b8a4a3
Yao Liu
682fe1e87d25524002b8a4a4
Weizhi Zhang
claimed_verified
2025-06-26T09:33:36.882Z
6667e801fd95ddf66cac84ff
/avatars/b75f6253160c81f558e6b40ed2ca1755.svg
Weizhi Zhang
user
WZDavid
682fe1e87d25524002b8a4a5
Qin Lu
682fe1e87d25524002b8a4a6
Liang Qiu
682fe1e87d25524002b8a4a7
Changlong Yu
682fe1e87d25524002b8a4a8
Puyang Xu
682fe1e87d25524002b8a4a9
Chao Zhang
682fe1e87d25524002b8a4aa
Bing Yin
682fe1e87d25524002b8a4ab
Hyokun Yun
682fe1e87d25524002b8a4ac
Lihong Li
WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement
  Learning
While reinforcement learning (RL) has demonstrated remarkable success in
enhancing large language models (LLMs), it has primarily focused on single-turn
tasks such as solving math problems. Training effective web agents for
multi-turn interactions remains challenging due to the complexity of
long-horizon decision-making across dynamic web interfaces. In this work, we
present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework
for training web agents. It learns directly from online interactions with web
environments by asynchronously generating diverse trajectories, entirely guided
by binary rewards depending on task success. Experiments on the WebArena-Lite
benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task
success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to
44.8%, significantly outperforming existing state-of-the-art methods and strong
proprietary models such as OpenAI o3. In-depth analyses reveal the
effectiveness of the thinking-based prompting strategy and test-time scaling
through increased interactions for web tasks. We further investigate different
RL initialization policies by introducing two variants, namely WebAgent-R1-Zero
and WebAgent-R1-CoT, which highlight the importance of the warm-up training
stage (i.e., behavior cloning) and provide insights on incorporating long
chain-of-thought (CoT) reasoning in web agents.
WebAgent-R1 is an RL framework for training web agents in multi-turn interactions, achieving high success rates compared to existing methods.
reinforcement learning large language models multi-turn interactions long-horizon decision-making web interfaces end-to-end multi-turn RL framework asymptotically generating diverse trajectories binary rewards WebArena-Lite benchmark Qwen-2.5-3B Llama-3.1-8B OpenAI o3 thinking-based prompting strategy test-time scaling RL initialization policies WebAgent-R1-Zero WebAgent-R1-CoT warm-up training stage behavior cloning long chain-of-thought reasoning