title: Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit
    Assignment
summary: This paper investigates approaches to enhance the reasoning capabilities of
  Large Language Model (LLM) agents using Reinforcement Learning (RL).
  Specifically, we focus on multi-turn tool-use scenarios, which can be naturally
  modeled as Markov Decision Processes (MDPs). While existing approaches often
  train multi-turn LLM agents with trajectory-level advantage estimation in
  bandit settings, they struggle with turn-level credit assignment across
  multiple decision steps, limiting their performance on multi-turn reasoning
  tasks. To address this, we introduce a fine-grained turn-level advantage
  estimation strategy to enable more precise credit assignment in multi-turn
  agent interactions. The strategy is general and can be incorporated into
  various RL algorithms such as Group Relative Preference Optimization (GRPO).
  Our experimental evaluation on multi-turn reasoning and search-based tool-use
  tasks with GRPO implementations highlights the effectiveness of the MDP
  framework and the turn-level credit assignment in advancing the multi-turn
  reasoning capabilities of LLM agents in complex decision-making settings. Our
  method achieves 100% success in tool execution and 50% accuracy in exact answer
  matching, significantly outperforming baselines, which fail to invoke tools and
  achieve only 20-30% exact match accuracy.
paper_authors._id: 6832def11f8965bd58214baf
paper_authors.name: Siliang Zeng
paper_authors._id: 6832def11f8965bd58214bb0
paper_authors.name: Quan Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T20:18:40.227Z
paper_authors._id: 662a037140363679757cc292
paper_authors.avatarUrl: /avatars/fc41455247cc0744f24ac9255293ddf5.svg
paper_authors.fullname: Quan Wei
paper_authors.type: user
paper_authors.user: quanwei0
paper_authors._id: 6832def11f8965bd58214bb1
paper_authors.name: William Brown
paper_authors._id: 6832def11f8965bd58214bb2
paper_authors.name: Oana Frunza
paper_authors._id: 6832def11f8965bd58214bb3
paper_authors.name: Yuriy Nevmyvaka
paper_authors._id: 6832def11f8965bd58214bb4
paper_authors.name: Mingyi Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:49:23.598Z
paper_authors._id: 679a5fa807076ced8ae5f41b
paper_authors.avatarUrl: /avatars/63cac0a05c9c7500c9ab9c75667a38d7.svg
paper_authors.fullname: Mingyi Hong
paper_authors.type: user
paper_authors.user: poborskypaul
paper_title: Reinforcing Multi-Turn Reasoning in LLM Agents via Turn-Level Credit
    Assignment
paper_summary: This paper investigates approaches to enhance the reasoning capabilities of
  Large Language Model (LLM) agents using Reinforcement Learning (RL).
  Specifically, we focus on multi-turn tool-use scenarios, which can be naturally
  modeled as Markov Decision Processes (MDPs). While existing approaches often
  train multi-turn LLM agents with trajectory-level advantage estimation in
  bandit settings, they struggle with turn-level credit assignment across
  multiple decision steps, limiting their performance on multi-turn reasoning
  tasks. To address this, we introduce a fine-grained turn-level advantage
  estimation strategy to enable more precise credit assignment in multi-turn
  agent interactions. The strategy is general and can be incorporated into
  various RL algorithms such as Group Relative Preference Optimization (GRPO).
  Our experimental evaluation on multi-turn reasoning and search-based tool-use
  tasks with GRPO implementations highlights the effectiveness of the MDP
  framework and the turn-level credit assignment in advancing the multi-turn
  reasoning capabilities of LLM agents in complex decision-making settings. Our
  method achieves 100% success in tool execution and 50% accuracy in exact answer
  matching, significantly outperforming baselines, which fail to invoke tools and
  achieve only 20-30% exact match accuracy.
paper_ai_summary: Reinforcement Learning with turn-level credit assignment enhances Large Language Model reasoning capabilities in multi-turn tool-use scenarios.
paper_ai_keywords: Large Language Model Reinforcement Learning multi-turn tool-use Markov Decision Processes trajectory-level advantage estimation turn-level credit assignment Group Relative Preference Optimization multi-turn reasoning search-based tool-use tool execution exact answer matching