title: RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn
    Reinforcement Learning
summary: Training large language models (LLMs) as interactive agents presents unique
  challenges including long-horizon decision making and interacting with
  stochastic environment feedback. While reinforcement learning (RL) has enabled
  progress in static tasks, multi-turn agent RL training remains underexplored.
  We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a
  general framework for trajectory-level agent RL, and introduce RAGEN, a modular
  system for training and evaluating LLM agents. Our study on three stylized
  environments reveals three core findings. First, our agent RL training shows a
  recurring mode of Echo Trap where reward variance cliffs and gradient spikes;
  we address this with StarPO-S, a stabilized variant with trajectory filtering,
  critic incorporation, and decoupled clipping. Second, we find the shaping of RL
  rollouts would benefit from diverse initial states, medium interaction
  granularity and more frequent sampling. Third, we show that without
  fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge
  through multi-turn RL and they may show shallow strategies or hallucinated
  thoughts. Code and environments are available at
  https://github.com/RAGEN-AI/RAGEN.
paper_authors._id: 68124aad34af9c89c46a800b
paper_authors.name: Zihan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:36:32.957Z
paper_authors._id: 63318e014bd187e9ce05cec6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63318e014bd187e9ce05cec6/rKN84BAXF_u0osSu6KJy5.jpeg
paper_authors.fullname: Zihan Wang
paper_authors.type: user
paper_authors.user: ZihanWang314
paper_authors._id: 68124aad34af9c89c46a800c
paper_authors.name: Kangrui Wang
paper_authors._id: 68124aad34af9c89c46a800d
paper_authors.name: Qineng Wang
paper_authors._id: 68124aad34af9c89c46a800e
paper_authors.name: Pingyue Zhang
paper_authors._id: 68124aad34af9c89c46a800f
paper_authors.name: Linjie Li
paper_authors._id: 68124aad34af9c89c46a8010
paper_authors.name: Zhengyuan Yang
paper_authors._id: 68124aad34af9c89c46a8011
paper_authors.name: Kefan Yu
paper_authors._id: 68124aad34af9c89c46a8012
paper_authors.name: Minh Nhat Nguyen
paper_authors._id: 68124aad34af9c89c46a8013
paper_authors.name: Licheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:36:30.696Z
paper_authors._id: 661caf2f6158c41091b3954e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661caf2f6158c41091b3954e/PZaPkeHQpcmv1cHK44Hew.jpeg
paper_authors.fullname: Licheng Liu
paper_authors.type: user
paper_authors.user: LichengLiu03
paper_authors._id: 68124aad34af9c89c46a8014
paper_authors.name: Eli Gottlieb
paper_authors._id: 68124aad34af9c89c46a8015
paper_authors.name: Monica Lam
paper_authors._id: 68124aad34af9c89c46a8016
paper_authors.name: Yiping Lu
paper_authors._id: 68124aad34af9c89c46a8017
paper_authors.name: Kyunghyun Cho
paper_authors._id: 68124aad34af9c89c46a8018
paper_authors.name: Jiajun Wu
paper_authors._id: 68124aad34af9c89c46a8019
paper_authors.name: Li Fei-Fei
paper_authors._id: 68124aad34af9c89c46a801a
paper_authors.name: Lijuan Wang
paper_authors._id: 68124aad34af9c89c46a801b
paper_authors.name: Yejin Choi
paper_authors._id: 68124aad34af9c89c46a801c
paper_authors.name: Manling Li
paper_title: RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn
    Reinforcement Learning
paper_summary: Training large language models (LLMs) as interactive agents presents unique
  challenges including long-horizon decision making and interacting with
  stochastic environment feedback. While reinforcement learning (RL) has enabled
  progress in static tasks, multi-turn agent RL training remains underexplored.
  We propose StarPO (State-Thinking-Actions-Reward Policy Optimization), a
  general framework for trajectory-level agent RL, and introduce RAGEN, a modular
  system for training and evaluating LLM agents. Our study on three stylized
  environments reveals three core findings. First, our agent RL training shows a
  recurring mode of Echo Trap where reward variance cliffs and gradient spikes;
  we address this with StarPO-S, a stabilized variant with trajectory filtering,
  critic incorporation, and decoupled clipping. Second, we find the shaping of RL
  rollouts would benefit from diverse initial states, medium interaction
  granularity and more frequent sampling. Third, we show that without
  fine-grained, reasoning-aware reward signals, agent reasoning hardly emerge
  through multi-turn RL and they may show shallow strategies or hallucinated
  thoughts. Code and environments are available at
  https://github.com/RAGEN-AI/RAGEN.
paper_ai_summary: StarPO and RAGEN address challenges in training interactive language model agents through RL, improving stability and reward shaping in diverse environments.
paper_ai_keywords: StarPO RAGEN trajectory-level agent RL Echo Trap trajectory filtering critic incorporation decoupled clipping RL rollouts initial states interaction granularity sampling reasoning-aware reward signals agent reasoning shallow strategies hallucinated thoughts