Thought Anchors: Which LLM Reasoning Steps Matter?
Reasoning large language models have recently achieved state-of-the-art
performance in many fields. However, their long-form chain-of-thought reasoning
creates interpretability challenges as each generated token depends on all
previous ones, making the computation harder to decompose. We argue that
analyzing reasoning traces at the sentence level is a promising approach to
understanding reasoning processes. We present three complementary attribution
methods: (1) a black-box method measuring each sentence's counterfactual
importance by comparing final answers across 100 rollouts conditioned on the
model generating that sentence or one with a different meaning; (2) a white-box
method of aggregating attention patterns between pairs of sentences, which
identified ``broadcasting'' sentences that receive disproportionate attention
from all future sentences via ``receiver'' attention heads; (3) a causal
attribution method measuring logical connections between sentences by
suppressing attention toward one sentence and measuring the effect on each
future sentence's tokens. Each method provides evidence for the existence of
thought anchors, reasoning steps that have outsized importance and that
disproportionately influence the subsequent reasoning process. These thought
anchors are typically planning or backtracking sentences. We provide an
open-source tool (www.thought-anchors.com) for visualizing the outputs of our
methods, and present a case study showing converging patterns across methods
that map how a model performs multi-step reasoning. The consistency across
methods demonstrates the potential of sentence-level analysis for a deeper
understanding of reasoning models.
685bd0ecaafa002dfd2bdef8
Paul C. Bogdan
claimed_verified
2025-06-27T08:57:44.222Z
6759711e45790f8a84a39ce5
/avatars/098052c8e44ca0c6edb3ce7171cfc753.svg
Paul Bogdan
user
pbogdan
685bd0ecaafa002dfd2bdef9
Uzay Macar
claimed_verified
2025-06-25T12:56:22.116Z
67166e4614c6788e479c3d60
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QuFz2ZIMbzNPcHe6uq7uL.jpeg
Uzay Macar
user
uzaymacar
685bd0ecaafa002dfd2bdefa
Neel Nanda
685bd0ecaafa002dfd2bdefb
Arthur Conmy
Thought Anchors: Which LLM Reasoning Steps Matter?
Reasoning large language models have recently achieved state-of-the-art
performance in many fields. However, their long-form chain-of-thought reasoning
creates interpretability challenges as each generated token depends on all
previous ones, making the computation harder to decompose. We argue that
analyzing reasoning traces at the sentence level is a promising approach to
understanding reasoning processes. We present three complementary attribution
methods: (1) a black-box method measuring each sentence's counterfactual
importance by comparing final answers across 100 rollouts conditioned on the
model generating that sentence or one with a different meaning; (2) a white-box
method of aggregating attention patterns between pairs of sentences, which
identified ``broadcasting'' sentences that receive disproportionate attention
from all future sentences via ``receiver'' attention heads; (3) a causal
attribution method measuring logical connections between sentences by
suppressing attention toward one sentence and measuring the effect on each
future sentence's tokens. Each method provides evidence for the existence of
thought anchors, reasoning steps that have outsized importance and that
disproportionately influence the subsequent reasoning process. These thought
anchors are typically planning or backtracking sentences. We provide an
open-source tool (www.thought-anchors.com) for visualizing the outputs of our
methods, and present a case study showing converging patterns across methods
that map how a model performs multi-step reasoning. The consistency across
methods demonstrates the potential of sentence-level analysis for a deeper
understanding of reasoning models.
Sentence-level attribution methods uncover critical thought anchors in large language models' reasoning processes, enhancing interpretability.
large language models chain-of-thought reasoning interpretability challenges black-box method counterfactual importance white-box method attention patterns broadcasting sentences receiver attention heads causal attribution method logical connections thought anchors planning sentences backtracking sentences