title: LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making
    Abilities
summary: The success of Large Language Models (LLMs) has sparked interest in various
  agentic applications. A key hypothesis is that LLMs, leveraging common sense
  and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently
  solve complex domains. However, LLM agents have been found to suffer from
  sub-optimal exploration and the knowing-doing gap, the inability to effectively
  act on knowledge present in the model. In this work, we systematically study
  why LLMs perform sub-optimally in decision-making scenarios. In particular, we
  closely examine three prevalent failure modes: greediness, frequency bias, and
  the knowing-doing gap. We propose mitigation of these shortcomings by
  fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales.
  Our experiments across multi-armed bandits, contextual bandits, and
  Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making
  abilities of LLMs by increasing exploration and narrowing the knowing-doing
  gap. Finally, we study both classic exploration mechanisms, such as
  epsilon-greedy, and LLM-specific approaches, such as self-correction and
  self-consistency, to enable more effective fine-tuning of LLMs for
  decision-making.
paper_authors._id: 68087a231e425a6eee93570d
paper_authors.name: Thomas Schmied
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:54:06.731Z
paper_authors._id: 64c3849269b1a6796052eac7
paper_authors.avatarUrl: /avatars/9f0c832d5b51b659c7bb83074f02a648.svg
paper_authors.fullname: Thomas Schmied
paper_authors.type: user
paper_authors.user: thomasschmied
paper_authors._id: 68087a231e425a6eee93570e
paper_authors.name: Jörg Bornschein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:54:12.909Z
paper_authors._id: 6679bdb03043e7ed261a7699
paper_authors.avatarUrl: /avatars/482ce2c4b2daf194161a0210e66d96de.svg
paper_authors.fullname: Jörg Bornschein
paper_authors.type: user
paper_authors.user: jbornschein
paper_authors._id: 68087a231e425a6eee93570f
paper_authors.name: Jordi Grau-Moya
paper_authors._id: 68087a231e425a6eee935710
paper_authors.name: Markus Wulfmeier
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:50:27.722Z
paper_authors._id: 680e82a9252bc7d0a3b1228d
paper_authors.avatarUrl: /avatars/3028d97c7702f259469a80cbb4ed13de.svg
paper_authors.fullname: Markus Wulfmeier
paper_authors.type: user
paper_authors.user: mwulfmeier
paper_authors._id: 68087a231e425a6eee935711
paper_authors.name: Razvan Pascanu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-23T13:54:25.503Z
paper_authors._id: 64b9310403124195cd9778ec
paper_authors.avatarUrl: /avatars/57c594d3d0f97d3010b15b6a0806451c.svg
paper_authors.fullname: Razvan Pascanu
paper_authors.type: user
paper_authors.user: razp
paper_title: LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making
    Abilities
paper_summary: The success of Large Language Models (LLMs) has sparked interest in various
  agentic applications. A key hypothesis is that LLMs, leveraging common sense
  and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently
  solve complex domains. However, LLM agents have been found to suffer from
  sub-optimal exploration and the knowing-doing gap, the inability to effectively
  act on knowledge present in the model. In this work, we systematically study
  why LLMs perform sub-optimally in decision-making scenarios. In particular, we
  closely examine three prevalent failure modes: greediness, frequency bias, and
  the knowing-doing gap. We propose mitigation of these shortcomings by
  fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales.
  Our experiments across multi-armed bandits, contextual bandits, and
  Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making
  abilities of LLMs by increasing exploration and narrowing the knowing-doing
  gap. Finally, we study both classic exploration mechanisms, such as
  epsilon-greedy, and LLM-specific approaches, such as self-correction and
  self-consistency, to enable more effective fine-tuning of LLMs for
  decision-making.
paper_ai_summary: Reinforcement learning fine-tuning with Chain-of-Thought rationales enhances Large Language Models' decision-making abilities by addressing greediness, frequency bias, and the knowing-doing gap.
paper_ai_keywords: Large Language Models Chain-of-Thought (CoT) decision-making sub-optimal exploration knowing-doing gap Reinforcement Learning (RL) multi-armed bandits contextual bandits Tic-tac-toe $\epsilon$-greedy self-correction self-consistency