Sparse Autoencoders Find Highly Interpretable Features in Language
  Models
One of the roadblocks to a better understanding of neural networks' internals
is polysemanticity, where neurons appear to activate in multiple,
semantically distinct contexts. Polysemanticity prevents us from identifying
concise, human-understandable explanations for what neural networks are doing
internally. One hypothesised cause of polysemanticity is
superposition, where neural networks represent more features than they
have neurons by assigning features to an overcomplete set of directions in
activation space, rather than to individual neurons. Here, we attempt to
identify those directions, using sparse autoencoders to reconstruct the
internal activations of a language model. These autoencoders learn sets of
sparsely activating features that are more interpretable and monosemantic than
directions identified by alternative approaches, where interpretability is
measured by automated methods. Ablating these features enables precise model
editing, for example, by removing capabilities such as pronoun prediction,
while disrupting model behaviour less than prior techniques. This work
indicates that it is possible to resolve superposition in language models using
a scalable, unsupervised method. Our method may serve as a foundation for
future mechanistic interpretability work, which we hope will enable greater
model transparency and steerability.
6507b936a7ba30bf62ccc74f
Hoagy Cunningham
admin_assigned
2023-09-18T10:18:29.959Z
636146e7472131c3bc538bd8
/avatars/9db880163cc0eea796165d8bf5e2a91f.svg
Hoagy Cunningham
user
HoagyC
6507b936a7ba30bf62ccc750
Aidan Ewart
admin_assigned
2023-09-18T10:18:38.015Z
643bbb729f5d314db2d89c22
https://cdn-avatars.huggingface.co/v1/production/uploads/643bbb729f5d314db2d89c22/bWxfTXlWY1y1-rS7Czqup.jpeg
Aidan Ewart
user
Baidicoot
6507b936a7ba30bf62ccc751
Logan Riggs
admin_assigned
2023-09-18T10:18:44.781Z
6366a684953957183c0fb596
/avatars/1a45039f224d78f126689a784f3f05f4.svg
Logan Riggs Smith
user
Elriggs
6507b936a7ba30bf62ccc752
Robert Huben
admin_assigned
2023-09-18T10:18:53.618Z
647e0050a49bffab5d7108c2
/avatars/707dbd12cc9d4b4711fd7e660385cd2d.svg
Robert Huben
user
RobertHuben
6507b936a7ba30bf62ccc753
Lee Sharkey
Sparse Autoencoders Find Highly Interpretable Features in Language
  Models
One of the roadblocks to a better understanding of neural networks' internals
is polysemanticity, where neurons appear to activate in multiple,
semantically distinct contexts. Polysemanticity prevents us from identifying
concise, human-understandable explanations for what neural networks are doing
internally. One hypothesised cause of polysemanticity is
superposition, where neural networks represent more features than they
have neurons by assigning features to an overcomplete set of directions in
activation space, rather than to individual neurons. Here, we attempt to
identify those directions, using sparse autoencoders to reconstruct the
internal activations of a language model. These autoencoders learn sets of
sparsely activating features that are more interpretable and monosemantic than
directions identified by alternative approaches, where interpretability is
measured by automated methods. Ablating these features enables precise model
editing, for example, by removing capabilities such as pronoun prediction,
while disrupting model behaviour less than prior techniques. This work
indicates that it is possible to resolve superposition in language models using
a scalable, unsupervised method. Our method may serve as a foundation for
future mechanistic interpretability work, which we hope will enable greater
model transparency and steerability.
Using sparse autoencoders, this work identifies and removes overcomplete directions in language model activations, improving interpretability and model editability.
polysemanticity superposition sparse autoencoders internal activations monosemantic interpretability model editing pronoun prediction