LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making
  Abilities
The success of Large Language Models (LLMs) has sparked interest in various
agentic applications. A key hypothesis is that LLMs, leveraging common sense
and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently
solve complex domains. However, LLM agents have been found to suffer from
sub-optimal exploration and the knowing-doing gap, the inability to effectively
act on knowledge present in the model. In this work, we systematically study
why LLMs perform sub-optimally in decision-making scenarios. In particular, we
closely examine three prevalent failure modes: greediness, frequency bias, and
the knowing-doing gap. We propose mitigation of these shortcomings by
fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales.
Our experiments across multi-armed bandits, contextual bandits, and
Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making
abilities of LLMs by increasing exploration and narrowing the knowing-doing
gap. Finally, we study both classic exploration mechanisms, such as
epsilon-greedy, and LLM-specific approaches, such as self-correction and
self-consistency, to enable more effective fine-tuning of LLMs for
decision-making.
68087a231e425a6eee93570d
Thomas Schmied
admin_assigned
2025-04-23T13:54:06.731Z
64c3849269b1a6796052eac7
/avatars/9f0c832d5b51b659c7bb83074f02a648.svg
Thomas Schmied
user
thomasschmied
68087a231e425a6eee93570e
Jörg Bornschein
admin_assigned
2025-04-23T13:54:12.909Z
6679bdb03043e7ed261a7699
/avatars/482ce2c4b2daf194161a0210e66d96de.svg
Jörg Bornschein
user
jbornschein
68087a231e425a6eee93570f
Jordi Grau-Moya
68087a231e425a6eee935710
Markus Wulfmeier
claimed_verified
2025-04-28T12:50:27.722Z
680e82a9252bc7d0a3b1228d
/avatars/3028d97c7702f259469a80cbb4ed13de.svg
Markus Wulfmeier
user
mwulfmeier
68087a231e425a6eee935711
Razvan Pascanu
admin_assigned
2025-04-23T13:54:25.503Z
64b9310403124195cd9778ec
/avatars/57c594d3d0f97d3010b15b6a0806451c.svg
Razvan Pascanu
user
razp
LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making
  Abilities
The success of Large Language Models (LLMs) has sparked interest in various
agentic applications. A key hypothesis is that LLMs, leveraging common sense
and Chain-of-Thought (CoT) reasoning, can effectively explore and efficiently
solve complex domains. However, LLM agents have been found to suffer from
sub-optimal exploration and the knowing-doing gap, the inability to effectively
act on knowledge present in the model. In this work, we systematically study
why LLMs perform sub-optimally in decision-making scenarios. In particular, we
closely examine three prevalent failure modes: greediness, frequency bias, and
the knowing-doing gap. We propose mitigation of these shortcomings by
fine-tuning via Reinforcement Learning (RL) on self-generated CoT rationales.
Our experiments across multi-armed bandits, contextual bandits, and
Tic-tac-toe, demonstrate that RL fine-tuning enhances the decision-making
abilities of LLMs by increasing exploration and narrowing the knowing-doing
gap. Finally, we study both classic exploration mechanisms, such as
epsilon-greedy, and LLM-specific approaches, such as self-correction and
self-consistency, to enable more effective fine-tuning of LLMs for
decision-making.
Reinforcement learning fine-tuning with Chain-of-Thought rationales enhances Large Language Models' decision-making abilities by addressing greediness, frequency bias, and the knowing-doing gap.
Large Language Models Chain-of-Thought (CoT) decision-making sub-optimal exploration knowing-doing gap Reinforcement Learning (RL) multi-armed bandits contextual bandits Tic-tac-toe $\epsilon$-greedy self-correction self-consistency