title: Interpreting Emergent Planning in Model-Free Reinforcement Learning
summary: We present the first mechanistic evidence that model-free reinforcement
  learning agents can learn to plan. This is achieved by applying a methodology
  based on concept-based interpretability to a model-free agent in Sokoban -- a
  commonly used benchmark for studying planning. Specifically, we demonstrate
  that DRC, a generic model-free agent introduced by Guez et al. (2019), uses
  learned concept representations to internally formulate plans that both predict
  the long-term effects of actions on the environment and influence action
  selection. Our methodology involves: (1) probing for planning-relevant
  concepts, (2) investigating plan formation within the agent's representations,
  and (3) verifying that discovered plans (in the agent's representations) have a
  causal effect on the agent's behavior through interventions. We also show that
  the emergence of these plans coincides with the emergence of a planning-like
  property: the ability to benefit from additional test-time compute. Finally, we
  perform a qualitative analysis of the planning algorithm learned by the agent
  and discover a strong resemblance to parallelized bidirectional search. Our
  findings advance understanding of the internal mechanisms underlying planning
  behavior in agents, which is important given the recent trend of emergent
  planning and reasoning capabilities in LLMs through RL
paper_authors._id: 67eea9e5117231f8bb04402b
paper_authors.name: Thomas Bush
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-03T19:20:34.085Z
paper_authors._id: 65d0c00b0954f06e472909f4
paper_authors.avatarUrl: /avatars/7dd76d922b781ed9895c7f4e62fefd9c.svg
paper_authors.fullname: tom bush
paper_authors.type: user
paper_authors.user: tuphs
paper_authors._id: 67eea9e5117231f8bb04402c
paper_authors.name: Stephen Chung
paper_authors._id: 67eea9e5117231f8bb04402d
paper_authors.name: Usman Anwar
paper_authors._id: 67eea9e5117231f8bb04402e
paper_authors.name: Adrià Garriga-Alonso
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-04-03T15:31:53.577Z
paper_authors._id: 645ecd18f0f92653b9f33d4e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645ecd18f0f92653b9f33d4e/nHDMWtM9ZHrji0c4Y4XW1.jpeg
paper_authors.fullname: Adrià Garriga-Alonso
paper_authors.type: user
paper_authors.user: agaralon
paper_authors._id: 67eea9e5117231f8bb04402f
paper_authors.name: David Krueger
paper_title: Interpreting Emergent Planning in Model-Free Reinforcement Learning
paper_summary: We present the first mechanistic evidence that model-free reinforcement
  learning agents can learn to plan. This is achieved by applying a methodology
  based on concept-based interpretability to a model-free agent in Sokoban -- a
  commonly used benchmark for studying planning. Specifically, we demonstrate
  that DRC, a generic model-free agent introduced by Guez et al. (2019), uses
  learned concept representations to internally formulate plans that both predict
  the long-term effects of actions on the environment and influence action
  selection. Our methodology involves: (1) probing for planning-relevant
  concepts, (2) investigating plan formation within the agent's representations,
  and (3) verifying that discovered plans (in the agent's representations) have a
  causal effect on the agent's behavior through interventions. We also show that
  the emergence of these plans coincides with the emergence of a planning-like
  property: the ability to benefit from additional test-time compute. Finally, we
  perform a qualitative analysis of the planning algorithm learned by the agent
  and discover a strong resemblance to parallelized bidirectional search. Our
  findings advance understanding of the internal mechanisms underlying planning
  behavior in agents, which is important given the recent trend of emergent
  planning and reasoning capabilities in LLMs through RL
paper_ai_summary: Model-free reinforcement learning agents can learn to plan using concept-based interpretability, showing long-term action prediction and influence on behavior, resembling parallelized bidirectional search.
paper_ai_keywords: model-free reinforcement learning planners concept-based interpretability Sokoban DRC concept representations plan formation causal effects planning-like property emergent planning reasoning capabilities LLMs RL