title: I Have Covered All the Bases Here: Interpreting Reasoning Features in
    Large Language Models via Sparse Autoencoders
summary: Large Language Models (LLMs) have achieved remarkable success in natural
  language processing. Recent advances have led to the developing of a new class
  of reasoning LLMs; for example, open-source DeepSeek-R1 has achieved
  state-of-the-art performance by integrating deep thinking and complex
  reasoning. Despite these impressive capabilities, the internal reasoning
  mechanisms of such models remain unexplored. In this work, we employ Sparse
  Autoencoders (SAEs), a method to learn a sparse decomposition of latent
  representations of a neural network into interpretable features, to identify
  features that drive reasoning in the DeepSeek-R1 series of models. First, we
  propose an approach to extract candidate ''reasoning features'' from SAE
  representations. We validate these features through empirical analysis and
  interpretability methods, demonstrating their direct correlation with the
  model's reasoning abilities. Crucially, we demonstrate that steering these
  features systematically enhances reasoning performance, offering the first
  mechanistic account of reasoning in LLMs. Code available at
  https://github.com/AIRI-Institute/SAE-Reasoning
paper_authors._id: 67e25fe88e6c927eb7794abd
paper_authors.name: Andrey Galichin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-28T12:24:57.801Z
paper_authors._id: 661e44cf1d8ffc49b57ba07e
paper_authors.avatarUrl: /avatars/3e937cc4f784b369b9f996ba82d1b81d.svg
paper_authors.fullname: Andrey Galichin
paper_authors.type: user
paper_authors.user: andreuka18
paper_authors._id: 67e25fe88e6c927eb7794abe
paper_authors.name: Alexey Dontsov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T08:18:43.467Z
paper_authors._id: 60cd95ee15ecba5f2200304a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60cd95ee15ecba5f2200304a/3gMYeWm8wQO5KfqE5RmEe.jpeg
paper_authors.fullname: Alexey Dontsov
paper_authors.type: user
paper_authors.user: therem
paper_authors._id: 67e25fe88e6c927eb7794abf
paper_authors.name: Polina Druzhinina
paper_authors._id: 67e25fe88e6c927eb7794ac0
paper_authors.name: Anton Razzhigaev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T09:05:58.409Z
paper_authors._id: 6172aaeec8e66e2aa84c06b9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6172aaeec8e66e2aa84c06b9/ZdRZSp3P1SU6CIDbvQwkv.jpeg
paper_authors.fullname: Anton Razzhigaev
paper_authors.type: user
paper_authors.user: razzant
paper_authors._id: 67e25fe88e6c927eb7794ac1
paper_authors.name: Oleg Y. Rogov
paper_authors._id: 67e25fe88e6c927eb7794ac2
paper_authors.name: Elena Tutubalina
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T09:05:56.401Z
paper_authors._id: 662f8d645c4db70c77a203b0
paper_authors.avatarUrl: /avatars/72f9a3c39b3ba5114388d16a35524835.svg
paper_authors.fullname: Elena Tutubalina
paper_authors.type: user
paper_authors.user: tlenusik
paper_authors._id: 67e25fe88e6c927eb7794ac3
paper_authors.name: Ivan Oseledets
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-25T19:02:24.076Z
paper_authors._id: 6169a581d05945bfd8718dfa
paper_authors.avatarUrl: /avatars/1892ab06a7ddb557232777de3cbec470.svg
paper_authors.fullname: Ivan Oseledets
paper_authors.type: user
paper_authors.user: oseledets
paper_title: I Have Covered All the Bases Here: Interpreting Reasoning Features in
    Large Language Models via Sparse Autoencoders
paper_summary: Large Language Models (LLMs) have achieved remarkable success in natural
  language processing. Recent advances have led to the developing of a new class
  of reasoning LLMs; for example, open-source DeepSeek-R1 has achieved
  state-of-the-art performance by integrating deep thinking and complex
  reasoning. Despite these impressive capabilities, the internal reasoning
  mechanisms of such models remain unexplored. In this work, we employ Sparse
  Autoencoders (SAEs), a method to learn a sparse decomposition of latent
  representations of a neural network into interpretable features, to identify
  features that drive reasoning in the DeepSeek-R1 series of models. First, we
  propose an approach to extract candidate ''reasoning features'' from SAE
  representations. We validate these features through empirical analysis and
  interpretability methods, demonstrating their direct correlation with the
  model's reasoning abilities. Crucially, we demonstrate that steering these
  features systematically enhances reasoning performance, offering the first
  mechanistic account of reasoning in LLMs. Code available at
  https://github.com/AIRI-Institute/SAE-Reasoning
paper_ai_summary: A study uses Sparse Autoencoders to identify and enhance reasoning features in Large Language Models, providing insights into the mechanisms driving their reasoning capabilities.
paper_ai_keywords: Sparse Autoencoders SAEs latent representations interpretable features reasoning features empirical analysis interpretability methods reasoning performance