Analyze Feature Flow to Enhance Interpretation and Steering in Language
  Models
We introduce a new approach to systematically map features discovered by
sparse autoencoder across consecutive layers of large language models,
extending earlier work that examined inter-layer feature links. By using a
data-free cosine similarity technique, we trace how specific features persist,
transform, or first appear at each stage. This method yields granular flow
graphs of feature evolution, enabling fine-grained interpretability and
mechanistic insights into model computations. Crucially, we demonstrate how
these cross-layer feature maps facilitate direct steering of model behavior by
amplifying or suppressing chosen features, achieving targeted thematic control
in text generation. Together, our findings highlight the utility of a causal,
cross-layer interpretability framework that not only clarifies how features
develop through forward passes but also provides new means for transparent
manipulation of large language models.
67a59c4e7ffacd843a56404a
Daniil Laptev
claimed_verified
2025-02-07T09:58:04.546Z
634c5f8cfb80cc6bcaf42c03
/avatars/1f37db0e70cbaf9707f4c8cbcee37ca0.svg
Daniil Laptev
user
dlaptev
67a59c4e7ffacd843a56404b
Nikita Balagansky
claimed_verified
2025-02-07T09:58:02.693Z
60b364e7f88532cd79eaff7b
https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
Nikita Balagansky
user
elephantmipt
67a59c4e7ffacd843a56404c
Yaroslav Aksenov
claimed_verified
2025-02-13T20:36:09.037Z
63ed5676684767daecac6f8a
/avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
Yaroslav Aksenov
user
yaraksen
67a59c4e7ffacd843a56404d
Daniil Gavrilov
claimed_verified
2025-02-07T09:58:06.718Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
Analyze Feature Flow to Enhance Interpretation and Steering in Language
  Models
We introduce a new approach to systematically map features discovered by
sparse autoencoder across consecutive layers of large language models,
extending earlier work that examined inter-layer feature links. By using a
data-free cosine similarity technique, we trace how specific features persist,
transform, or first appear at each stage. This method yields granular flow
graphs of feature evolution, enabling fine-grained interpretability and
mechanistic insights into model computations. Crucially, we demonstrate how
these cross-layer feature maps facilitate direct steering of model behavior by
amplifying or suppressing chosen features, achieving targeted thematic control
in text generation. Together, our findings highlight the utility of a causal,
cross-layer interpretability framework that not only clarifies how features
develop through forward passes but also provides new means for transparent
manipulation of large language models.
A new data-free cosine similarity method maps and steers features across layers of large language models, enhancing interpretability and targeted control in text generation.
sparse autoencoder large language models data-free cosine similarity feature evolution forward passes