title: WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum
    Reinforcement Learning
summary: Large language models (LLMs) have shown remarkable potential as autonomous
  agents, particularly in web-based tasks. However, existing LLM web agents
  heavily rely on expensive proprietary LLM APIs, while open LLMs lack the
  necessary decision-making capabilities. This paper introduces WebRL, a
  self-evolving online curriculum reinforcement learning framework designed to
  train high-performance web agents using open LLMs. WebRL addresses three key
  challenges in building LLM web agents, including the scarcity of training
  tasks, sparse feedback signals, and policy distribution drift in online
  learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that
  generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised
  reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure
  consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4
  models into proficient web agents. On WebArena-Lite, WebRL improves the success
  rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B.
  These open models significantly surpass the performance of GPT-4-Turbo (17.6%)
  and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained
  on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's
  effectiveness in bridging the gap between open and proprietary LLM-based web
  agents, paving the way for more accessible and powerful autonomous web
  interaction systems.
paper_authors._id: 67299dd8ff3de35f30a2bffe
paper_authors.name: Zehan Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:58:00.956Z
paper_authors._id: 650ace9fa80a948447812fc5
paper_authors.avatarUrl: /avatars/e01fe0d973008f5e3cd3c99f895b5946.svg
paper_authors.fullname: zehanqi
paper_authors.type: user
paper_authors.user: zehanqi
paper_authors._id: 67299dd8ff3de35f30a2bfff
paper_authors.name: Xiao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:58:45.111Z
paper_authors._id: 62f098602ca4d32a7cd87aba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: ShawLiu
paper_authors._id: 67299dd8ff3de35f30a2c000
paper_authors.name: Iat Long Iong
paper_authors._id: 67299dd8ff3de35f30a2c001
paper_authors.name: Hanyu Lai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:56:47.720Z
paper_authors._id: 62ea2d48b493272c269e8f34
paper_authors.avatarUrl: /avatars/89b215dafa503b51ab212a9b63c82aca.svg
paper_authors.fullname: Hanyu Lai
paper_authors.type: user
paper_authors.user: hanyullai
paper_authors._id: 67299dd8ff3de35f30a2c002
paper_authors.name: Xueqiao Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:56:56.205Z
paper_authors._id: 6729afc3c519b000f45d053b
paper_authors.avatarUrl: /avatars/1d47cfd495e72316420fc16f7286df2b.svg
paper_authors.fullname: Xueqiao Sun
paper_authors.type: user
paper_authors.user: snow1007
paper_authors._id: 67299dd8ff3de35f30a2c003
paper_authors.name: Xinyue Yang
paper_authors._id: 67299dd8ff3de35f30a2c004
paper_authors.name: Jiadai Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:57:27.980Z
paper_authors._id: 64e82f2c9a3cd93b370e9936
paper_authors.avatarUrl: /avatars/942e6008e8e78583c2750d678fcaadb1.svg
paper_authors.fullname: jiadaisun
paper_authors.type: user
paper_authors.user: jiadaisun
paper_authors._id: 67299dd8ff3de35f30a2c005
paper_authors.name: Yu Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:57:33.794Z
paper_authors._id: 60a53adbf9b53404e7806277
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1621441193991-noauth.jpeg
paper_authors.fullname: Yu Yang
paper_authors.type: user
paper_authors.user: yuyang
paper_authors._id: 67299dd8ff3de35f30a2c006
paper_authors.name: Shuntian Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T13:57:53.493Z
paper_authors._id: 6464c91ef4f3102bb6c7418a
paper_authors.avatarUrl: /avatars/1e6ce129d8cd51e81d517e23ad9aee43.svg
paper_authors.fullname: Shuntian Yao
paper_authors.type: user
paper_authors.user: TMIP
paper_authors._id: 67299dd8ff3de35f30a2c007
paper_authors.name: Tianjie Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T07:57:23.819Z
paper_authors._id: 64dceda0f44d2407e60179f8
paper_authors.avatarUrl: /avatars/726d5ed318b40ae886b7387a532217e4.svg
paper_authors.fullname: Tianjie Zhang
paper_authors.type: user
paper_authors.user: tianjiezhang
paper_authors._id: 67299dd8ff3de35f30a2c008
paper_authors.name: Wei Xu
paper_authors._id: 67299dd8ff3de35f30a2c009
paper_authors.name: Jie Tang
paper_authors._id: 67299dd8ff3de35f30a2c00a
paper_authors.name: Yuxiao Dong
paper_title: WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum
    Reinforcement Learning
paper_summary: Large language models (LLMs) have shown remarkable potential as autonomous
  agents, particularly in web-based tasks. However, existing LLM web agents
  heavily rely on expensive proprietary LLM APIs, while open LLMs lack the
  necessary decision-making capabilities. This paper introduces WebRL, a
  self-evolving online curriculum reinforcement learning framework designed to
  train high-performance web agents using open LLMs. WebRL addresses three key
  challenges in building LLM web agents, including the scarcity of training
  tasks, sparse feedback signals, and policy distribution drift in online
  learning. Specifically, WebRL incorporates 1) a self-evolving curriculum that
  generates new tasks from unsuccessful attempts, 2) a robust outcome-supervised
  reward model (ORM), and 3) adaptive reinforcement learning strategies to ensure
  consistent improvements. We apply WebRL to transform open Llama-3.1 and GLM-4
  models into proficient web agents. On WebArena-Lite, WebRL improves the success
  rate of Llama-3.1-8B from 4.8% to 42.4%, and from 6.1% to 43% for GLM-4-9B.
  These open models significantly surpass the performance of GPT-4-Turbo (17.6%)
  and GPT-4o (13.9%) and outperform previous state-of-the-art web agents trained
  on open LLMs (AutoWebGLM, 18.2%). Our findings demonstrate WebRL's
  effectiveness in bridging the gap between open and proprietary LLM-based web
  agents, paving the way for more accessible and powerful autonomous web
  interaction systems.
paper_ai_summary: WebRL enhances open LLMs as web agents through a self-evolving curriculum, robust outcome-supervised reward model, and adaptive reinforcement learning strategies, achieving high success rates compared to proprietary models.
paper_ai_keywords: reinforcement learning self-evolving curriculum outcome-supervised reward model adaptive reinforcement learning web agents WebRL Llama-3.1 GLM-4 WebArena-Lite GPT-4-Turbo GPT-4o AutoWebGLM