title: WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement
    Learning
summary: While reinforcement learning (RL) has demonstrated remarkable success in
  enhancing large language models (LLMs), it has primarily focused on single-turn
  tasks such as solving math problems. Training effective web agents for
  multi-turn interactions remains challenging due to the complexity of
  long-horizon decision-making across dynamic web interfaces. In this work, we
  present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework
  for training web agents. It learns directly from online interactions with web
  environments by asynchronously generating diverse trajectories, entirely guided
  by binary rewards depending on task success. Experiments on the WebArena-Lite
  benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task
  success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to
  44.8%, significantly outperforming existing state-of-the-art methods and strong
  proprietary models such as OpenAI o3. In-depth analyses reveal the
  effectiveness of the thinking-based prompting strategy and test-time scaling
  through increased interactions for web tasks. We further investigate different
  RL initialization policies by introducing two variants, namely WebAgent-R1-Zero
  and WebAgent-R1-CoT, which highlight the importance of the warm-up training
  stage (i.e., behavior cloning) and provide insights on incorporating long
  chain-of-thought (CoT) reasoning in web agents.
paper_authors._id: 682fe1e87d25524002b8a4a1
paper_authors.name: Zhepei Wei
paper_authors._id: 682fe1e87d25524002b8a4a2
paper_authors.name: Wenlin Yao
paper_authors._id: 682fe1e87d25524002b8a4a3
paper_authors.name: Yao Liu
paper_authors._id: 682fe1e87d25524002b8a4a4
paper_authors.name: Weizhi Zhang
paper_authors._id: 682fe1e87d25524002b8a4a5
paper_authors.name: Qin Lu
paper_authors._id: 682fe1e87d25524002b8a4a6
paper_authors.name: Liang Qiu
paper_authors._id: 682fe1e87d25524002b8a4a7
paper_authors.name: Changlong Yu
paper_authors._id: 682fe1e87d25524002b8a4a8
paper_authors.name: Puyang Xu
paper_authors._id: 682fe1e87d25524002b8a4a9
paper_authors.name: Chao Zhang
paper_authors._id: 682fe1e87d25524002b8a4aa
paper_authors.name: Bing Yin
paper_authors._id: 682fe1e87d25524002b8a4ab
paper_authors.name: Hyokun Yun
paper_authors._id: 682fe1e87d25524002b8a4ac
paper_authors.name: Lihong Li
paper_title: WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement
    Learning
paper_summary: While reinforcement learning (RL) has demonstrated remarkable success in
  enhancing large language models (LLMs), it has primarily focused on single-turn
  tasks such as solving math problems. Training effective web agents for
  multi-turn interactions remains challenging due to the complexity of
  long-horizon decision-making across dynamic web interfaces. In this work, we
  present WebAgent-R1, a simple yet effective end-to-end multi-turn RL framework
  for training web agents. It learns directly from online interactions with web
  environments by asynchronously generating diverse trajectories, entirely guided
  by binary rewards depending on task success. Experiments on the WebArena-Lite
  benchmark demonstrate the effectiveness of WebAgent-R1, boosting the task
  success rate of Qwen-2.5-3B from 6.1% to 33.9% and Llama-3.1-8B from 8.5% to
  44.8%, significantly outperforming existing state-of-the-art methods and strong
  proprietary models such as OpenAI o3. In-depth analyses reveal the
  effectiveness of the thinking-based prompting strategy and test-time scaling
  through increased interactions for web tasks. We further investigate different
  RL initialization policies by introducing two variants, namely WebAgent-R1-Zero
  and WebAgent-R1-CoT, which highlight the importance of the warm-up training
  stage (i.e., behavior cloning) and provide insights on incorporating long
  chain-of-thought (CoT) reasoning in web agents.
paper_ai_summary: WebAgent-R1 is an RL framework for training web agents in multi-turn interactions, achieving high success rates compared to existing methods.
paper_ai_keywords: reinforcement learning large language models multi-turn interactions long-horizon decision-making web interfaces end-to-end multi-turn RL framework asymptotically generating diverse trajectories binary rewards WebArena-Lite benchmark Qwen-2.5-3B Llama-3.1-8B OpenAI o3 thinking-based prompting strategy test-time scaling RL initialization policies WebAgent-R1-Zero WebAgent-R1-CoT warm-up training stage behavior cloning long chain-of-thought reasoning