title: SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning
    Tasks
summary: Large language model (LLM) agents need to perform multi-turn interactions in
  real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM
  agents fail to perform effective credit assignment over multiple turns while
  leveraging the generalization capabilities of LLMs and it remains unclear how
  to develop such algorithms. To study this, we first introduce a new benchmark,
  ColBench, where an LLM agent interacts with a human collaborator over multiple
  turns to solve realistic tasks in backend programming and frontend design.
  Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with
  Step-WisE Evaluation from Training-time information), that uses a carefully
  designed optimization objective to train a critic model with access to
  additional training-time information. The critic provides step-level rewards
  for improving the policy model. Our experiments demonstrate that SWEET-RL
  achieves a 6% absolute improvement in success and win rates on ColBench
  compared to other state-of-the-art multi-turn RL algorithms, enabling
  Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic
  collaborative content creation.
paper_authors._id: 67db96db0d13633dcbe53fd6
paper_authors.name: Yifei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:29:41.066Z
paper_authors._id: 64fd0229e0dc35986bd3c0e5
paper_authors.avatarUrl: /avatars/94f5698f9104dad7288edb4460026fd8.svg
paper_authors.fullname: Yifei Zhou
paper_authors.type: user
paper_authors.user: yifeizhou
paper_authors._id: 67db96db0d13633dcbe53fd7
paper_authors.name: Song Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:30:15.739Z
paper_authors._id: 629ff655198a9a5a7562d437
paper_authors.avatarUrl: /avatars/42fd229a1521dc218c1fb385dd867b3b.svg
paper_authors.fullname: Song Jiang
paper_authors.type: user
paper_authors.user: songjiang
paper_authors._id: 67db96db0d13633dcbe53fd8
paper_authors.name: Yuandong Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:30:08.515Z
paper_authors._id: 6344cf73ee1504dbcd5bdfe7
paper_authors.avatarUrl: /avatars/6dd2bf1f9c5679e5c8c85d62c9836aac.svg
paper_authors.fullname: Yuandong Tian
paper_authors.type: user
paper_authors.user: tydsh
paper_authors._id: 67db96db0d13633dcbe53fd9
paper_authors.name: Jason Weston
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:30:02.617Z
paper_authors._id: 62f023a36a027498eaa2f9cc
paper_authors.avatarUrl: /avatars/8ac1c5c74d0957e3c6cc94b3a7795c37.svg
paper_authors.fullname: Jason Weston
paper_authors.type: user
paper_authors.user: spermwhale
paper_authors._id: 67db96db0d13633dcbe53fda
paper_authors.name: Sergey Levine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:29:54.687Z
paper_authors._id: 665ce54120a307a3754849dd
paper_authors.avatarUrl: /avatars/e698726e9be61dd50ce2efe372ed5dac.svg
paper_authors.fullname: Sergey Levine
paper_authors.type: user
paper_authors.user: svlevine
paper_authors._id: 67db96db0d13633dcbe53fdb
paper_authors.name: Sainbayar Sukhbaatar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-20T19:29:48.384Z
paper_authors._id: 66a8611eb51510d82ed54231
paper_authors.avatarUrl: /avatars/ad559e774fee4914091b82c9831ae2a2.svg
paper_authors.fullname: Sainbayar Sukhbaatar
paper_authors.type: user
paper_authors.user: sainbar
paper_authors._id: 67db96db0d13633dcbe53fdc
paper_authors.name: Xian Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:41:42.399Z
paper_authors._id: 659a395421a7431643caedda
paper_authors.avatarUrl: /avatars/c1e0bbcedce68fe3b4fe39e0cf01c65c.svg
paper_authors.fullname: Xian Li
paper_authors.type: user
paper_authors.user: xlxxl
paper_title: SWEET-RL: Training Multi-Turn LLM Agents on Collaborative Reasoning
    Tasks
paper_summary: Large language model (LLM) agents need to perform multi-turn interactions in
  real-world tasks. However, existing multi-turn RL algorithms for optimizing LLM
  agents fail to perform effective credit assignment over multiple turns while
  leveraging the generalization capabilities of LLMs and it remains unclear how
  to develop such algorithms. To study this, we first introduce a new benchmark,
  ColBench, where an LLM agent interacts with a human collaborator over multiple
  turns to solve realistic tasks in backend programming and frontend design.
  Building on this benchmark, we propose a novel RL algorithm, SWEET-RL (RL with
  Step-WisE Evaluation from Training-time information), that uses a carefully
  designed optimization objective to train a critic model with access to
  additional training-time information. The critic provides step-level rewards
  for improving the policy model. Our experiments demonstrate that SWEET-RL
  achieves a 6% absolute improvement in success and win rates on ColBench
  compared to other state-of-the-art multi-turn RL algorithms, enabling
  Llama-3.1-8B to match or exceed the performance of GPT4-o in realistic
  collaborative content creation.
paper_ai_summary: A novel RL algorithm, SWEET-RL, uses step-level rewards to enhance multi-turn interactions in LLMs, outperforming existing methods on a new benchmark for collaborative content creation tasks.
paper_ai_keywords: LLM multi-turn RL credit assignment ColBench backend programming frontend design critic model training-time information step-level rewards policy model success rates win rates GPT4-o