Decomposing MLP Activations into Interpretable Features via
  Semi-Nonnegative Matrix Factorization
A central goal for mechanistic interpretability has been to identify the
right units of analysis in large language models (LLMs) that causally explain
their outputs. While early work focused on individual neurons, evidence that
neurons often encode multiple concepts has motivated a shift toward analyzing
directions in activation space. A key question is how to find directions that
capture interpretable features in an unsupervised manner. Current methods rely
on dictionary learning with sparse autoencoders (SAEs), commonly trained over
residual stream activations to learn directions from scratch. However, SAEs
often struggle in causal evaluations and lack intrinsic interpretability, as
their learning is not explicitly tied to the computations of the model. Here,
we tackle these limitations by directly decomposing MLP activations with
semi-nonnegative matrix factorization (SNMF), such that the learned features
are (a) sparse linear combinations of co-activated neurons, and (b) mapped to
their activating inputs, making them directly interpretable. Experiments on
Llama 3.1, Gemma 2 and GPT-2 show that SNMF derived features outperform SAEs
and a strong supervised baseline (difference-in-means) on causal steering,
while aligning with human-interpretable concepts. Further analysis reveals that
specific neuron combinations are reused across semantically-related features,
exposing a hierarchical structure in the MLP's activation space. Together,
these results position SNMF as a simple and effective tool for identifying
interpretable features and dissecting concept representations in LLMs.
684c39883b733ba333687278
Or Shafran
claimed_verified
2025-06-13T14:59:09.236Z
67b738790d11464dcbc8adf5
/avatars/f59109ff562d7b037be2f81bb11df270.svg
Or Shafran
user
ordavids1
684c39883b733ba333687279
Atticus Geiger
extracted_confirmed
2025-06-13T15:21:27.694Z
627b2d0527dc4650b62eef42
/avatars/e70381850f5657b54e90f5539f3d74eb.svg
Atticus Geiger
user
atticusg
684c39883b733ba33368727a
Mor Geva
claimed_verified
2025-06-14T06:24:29.773Z
610b729f9da682cd54ad9adf
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140189042-noauth.jpeg
Mor Geva
user
mega
Decomposing MLP Activations into Interpretable Features via
  Semi-Nonnegative Matrix Factorization
A central goal for mechanistic interpretability has been to identify the
right units of analysis in large language models (LLMs) that causally explain
their outputs. While early work focused on individual neurons, evidence that
neurons often encode multiple concepts has motivated a shift toward analyzing
directions in activation space. A key question is how to find directions that
capture interpretable features in an unsupervised manner. Current methods rely
on dictionary learning with sparse autoencoders (SAEs), commonly trained over
residual stream activations to learn directions from scratch. However, SAEs
often struggle in causal evaluations and lack intrinsic interpretability, as
their learning is not explicitly tied to the computations of the model. Here,
we tackle these limitations by directly decomposing MLP activations with
semi-nonnegative matrix factorization (SNMF), such that the learned features
are (a) sparse linear combinations of co-activated neurons, and (b) mapped to
their activating inputs, making them directly interpretable. Experiments on
Llama 3.1, Gemma 2 and GPT-2 show that SNMF derived features outperform SAEs
and a strong supervised baseline (difference-in-means) on causal steering,
while aligning with human-interpretable concepts. Further analysis reveals that
specific neuron combinations are reused across semantically-related features,
exposing a hierarchical structure in the MLP's activation space. Together,
these results position SNMF as a simple and effective tool for identifying
interpretable features and dissecting concept representations in LLMs.
SNMF is used to identify interpretable features in LLMs by directly decomposing MLP activations, outperforming SAEs and supervised methods in causal evaluations and aligning with human-interpretable concepts.
mechanistic interpretability large language models (LLMs) individual neurons activation space dictionary learning sparse autoencoders (SAEs) semi-nonnegative matrix factorization (SNMF) MLP activations causal steering concept representations