Interpreting Emergent Planning in Model-Free Reinforcement Learning
We present the first mechanistic evidence that model-free reinforcement
learning agents can learn to plan. This is achieved by applying a methodology
based on concept-based interpretability to a model-free agent in Sokoban -- a
commonly used benchmark for studying planning. Specifically, we demonstrate
that DRC, a generic model-free agent introduced by Guez et al. (2019), uses
learned concept representations to internally formulate plans that both predict
the long-term effects of actions on the environment and influence action
selection. Our methodology involves: (1) probing for planning-relevant
concepts, (2) investigating plan formation within the agent's representations,
and (3) verifying that discovered plans (in the agent's representations) have a
causal effect on the agent's behavior through interventions. We also show that
the emergence of these plans coincides with the emergence of a planning-like
property: the ability to benefit from additional test-time compute. Finally, we
perform a qualitative analysis of the planning algorithm learned by the agent
and discover a strong resemblance to parallelized bidirectional search. Our
findings advance understanding of the internal mechanisms underlying planning
behavior in agents, which is important given the recent trend of emergent
planning and reasoning capabilities in LLMs through RL
67eea9e5117231f8bb04402b
Thomas Bush
claimed_verified
2025-04-03T19:20:34.085Z
65d0c00b0954f06e472909f4
/avatars/7dd76d922b781ed9895c7f4e62fefd9c.svg
tom bush
user
tuphs
67eea9e5117231f8bb04402c
Stephen Chung
67eea9e5117231f8bb04402d
Usman Anwar
67eea9e5117231f8bb04402e
Adrià Garriga-Alonso
extracted_pending
2025-04-03T15:31:53.577Z
645ecd18f0f92653b9f33d4e
https://cdn-avatars.huggingface.co/v1/production/uploads/645ecd18f0f92653b9f33d4e/nHDMWtM9ZHrji0c4Y4XW1.jpeg
Adrià Garriga-Alonso
user
agaralon
67eea9e5117231f8bb04402f
David Krueger
Interpreting Emergent Planning in Model-Free Reinforcement Learning
We present the first mechanistic evidence that model-free reinforcement
learning agents can learn to plan. This is achieved by applying a methodology
based on concept-based interpretability to a model-free agent in Sokoban -- a
commonly used benchmark for studying planning. Specifically, we demonstrate
that DRC, a generic model-free agent introduced by Guez et al. (2019), uses
learned concept representations to internally formulate plans that both predict
the long-term effects of actions on the environment and influence action
selection. Our methodology involves: (1) probing for planning-relevant
concepts, (2) investigating plan formation within the agent's representations,
and (3) verifying that discovered plans (in the agent's representations) have a
causal effect on the agent's behavior through interventions. We also show that
the emergence of these plans coincides with the emergence of a planning-like
property: the ability to benefit from additional test-time compute. Finally, we
perform a qualitative analysis of the planning algorithm learned by the agent
and discover a strong resemblance to parallelized bidirectional search. Our
findings advance understanding of the internal mechanisms underlying planning
behavior in agents, which is important given the recent trend of emergent
planning and reasoning capabilities in LLMs through RL
Model-free reinforcement learning agents can learn to plan using concept-based interpretability, showing long-term action prediction and influence on behavior, resembling parallelized bidirectional search.
model-free reinforcement learning planners concept-based interpretability Sokoban DRC concept representations plan formation causal effects planning-like property emergent planning reasoning capabilities LLMs RL