## Disentangling Dense Embeddings with Sparse Autoencoders

## Charles O'Neill ∗

Christine Ye ∗

The Australian National University charles.oneill@anu.edu.au

## Kartheik Iyer

Columbia University kartheikiyer@gmail.com

Stanford University cye@stanford.edu

## John F. Wu

Space Telescope Science Institute Johns Hopkins University jfwu@stsci.edu

## Abstract

Sparse autoencoders (SAEs) have shown promise in extracting interpretable features from complex neural networks. We present one of the first applications of SAEs to dense text embeddings from large language models, demonstrating their effectiveness in disentangling semantic concepts. By training SAEs on embeddings of over 420,000 scientific paper abstracts from computer science and astronomy, we show that the resulting sparse representations maintain semantic fidelity while offering interpretability. We analyse these learned features, exploring their behaviour across different model capacities and introducing a novel method for identifying 'feature families' that represent related concepts at varying levels of abstraction. To demonstrate the practical utility of our approach, we show how these interpretable features can be used to precisely steer semantic search, allowing for fine-grained control over query semantics. This work bridges the gap between the semantic richness of dense embeddings and the interpretability of sparse representations. We open source our embeddings, trained sparse autoencoders, and interpreted features, as well as a web app for exploring them.

## 1 Introduction

The advent of large language models has revolutionised natural language processing, enabling the representation of text in rich semantic spaces (Devlin et al., 2018; Brown et al., 2020). These dense neural vector embeddings capture nuanced semantic relationships, significantly enhancing downstream applications such as information retrieval (IR) and semantic search (Reimers et al., 2019; Gao et al., 2022; Wang et al., 2024). However, the power of these representations comes at a cost: reduced interpretability and limited user control, presenting significant challenges for fine-tuning and explaining search results (Cao et al., 2023a).

To address these limitations, recent research has explored methods to disentangle and interpret the information encoded in dense representations (Trifonov et al., 2018). Sparse autoencoders (SAEs) have emerged as a promising solution for extracting interpretable features from high-dimensional representations (Ng et al., 2011; Makhzani et al., 2013). By learning to reconstruct inputs as linear combinations of features in a higher-dimensional sparse basis, SAEs can disentangle complex representations into individually interpretable components. This approach has shown success in analysing and steering generative models (Conmy et al., 2024; Lee, 2024; Cunningham et al., 2023b), but its application to dense text embeddings remains unexplored.

∗ Joint first author contribution.

In this work, we present the first application of SAEs to dense text embeddings derived from large language models. We demonstrate that this approach bridges the gap between the semantic richness of dense embeddings and the interpretability of sparse representations, offering new possibilities for understanding and manipulating textual semantic spaces. In a direct demonstration of their utility for semantic search, we show how SAE features can be used to steer search results, which has been previously been applied to decoder-only transformers and diffusion models for guided generation (Elhage et al., 2022b; Daujotas, 2024). By causally manipulating features in the SAE's hidden representation of an embedding vector, we can perform precise adjustments of the semantic meaning of the vector upon reconstruction.

Our research makes the following key contributions:

- 1. We train SAEs with varying sizes on embeddings from a large corpus of scientific papers, demonstrating their effectiveness in learning interpretable features from dense text representations.
- 2. Weconduct a comprehensive analysis of the learned features, examining their interpretability, behaviour across different model capacities, and semantic properties.
- 3. We introduce the concept of SAE 'feature families', clusters of related features that allow for multi-scale semantic analysis and manipulation, and look at how features 'split' across levels of abstraction.
- 4. We demonstrate the practical utility of our approach by applying these interpretable features to enhance semantic search, allowing for fine-grained control over query semantics. We develop and open-source this as a tool that implements our SAE-enhanced semantic search system, as well as open-sourcing the underlying SAEs.

This paper is organised as follows: Section 2 provides background on embedding representations and sparse autoencoders. Section 3 details our methodology for training SAEs on text embeddings and analysing the resulting features, and the interpretability metrics of the learned features, as well as scaling behaviour of features across model capacities. Section 4 outlines our identification of feature families and feature splitting. Section 5 demonstrates the application of our approach to semantic search. Finally, Section 6 discusses the implications of our work and potential future directions.

## 2 Background and Related work

Whilst dense embeddings have dramatically improved performance across various NLP tasks, they present significant challenges in terms of interpretability.

## 2.1 Embeddings and Representation Learning

The evolution of word representations in NLP has progressed from simple one-hot encodings to sophisticated dense vector embeddings, each stage offering improvements in semantic expressiveness and contextual understanding. Early distributional semantics models like Latent Semantic Analysis (Deerwester et al., 1990) paved the way for prediction-based word embeddings such as Word2Vec (Mikolov et al., 2013a) and GloVe (Pennington et al., 2014), which capture semantic and syntactic relationships. The advent of contextualised word representations, such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018), marked a significant leap by generating dynamic word representations based on surrounding context. Recent developments have extended to sentence and document-level embeddings, with models like Sentence-BERT (Reimers et al., 2019) proving particularly effective for tasks such as semantic search and information retrieval (Gao et al., 2021).

While dense embeddings have dramatically improved the performance of NLP systems, they present challenges in terms of interpretability and fine-grained control. The high-dimensional, continuous nature of these representations makes it difficult to understand or manipulate specific semantic aspects encoded within them (Liu et al., 2019). This opacity can be particularly problematic in applications where explainability or precise semantic control is critical. Semantic search with dense embeddings has largely replaced traditional keyword search (Manning et al., 2008; Baeza-Yates et al., 1999; Furnas et al., 1987; Mikolov et al., 2013b; Devlin et al., 2018; Reimers et al., 2019), but faces ongoing challenges, including the 'curse of dense retrieval' (Reimers et al., 2022) - a phenomenon where the performance of dense retrieval methods degrades more rapidly than sparse methods as index size

Figure 1: Training and feature labelling process for our sparse autoencoder (SAE). The SAE is trained to minimise reconstruction loss on embeddings from astronomy and computer science paper abstracts. Each feature corresponds to a column in the decoder matrix, representing a direction in embedding space. Feature interpretation involves two steps: (1) An Interpreter language model identifies topics present in text that activates each feature but absent in non-activating text. (2) A separate Predictor language model assesses feature interpretability by stating its confidence that the feature will activate on unseen text, with confidence correlated with ground truth activations to quantify interpretability.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-5039eb80-d882-42ee-b528-70c2ab91d232.jpg)

increases - and lack of interpretability, which hinder fine-tuning of search results (Cao et al., 2023b; Turian et al., 2010).

## 2.2 Sparse autoencoders

In large language models, the superposition hypothesis suggests that dense neural networks are highly underparameterised, and perform computations involving many more concepts than neurons (Elhage et al., 2022a). Because these semantic concepts, or features , are quite sparse, models compensate encoding multiple features within the same set of neurons. However, this also leads to complex, overlapping representations that are difficult to interpret on a single-neuron basis. Similarly, in embedding spaces, features are not represented monosemantically in individual dimensions. Instead, each feature is typically distributed across multiple dimensions, and conversely, each dimension may contribute to the representation of multiple features. This distributed representation allows embedding models to efficiently encode a large number of features in a relatively low-dimensional space, but it also makes the embeddings challenging to interpret directly.

To address this challenge, sparse autoencoders (SAEs) have emerged as a promising solution. SAEs learn to reconstruct inputs using a sparse set of features in a higher-dimensional space, potentially disentangling superposed features (Elhage et al., 2022b; Olshausen et al., 1997). By encouraging this disentanglement, SAEs aim to reveal more interpretable and semantically meaningful representations, demonstrating efficacy in uncovering interpretable features in large language model activations (Donoho, 2006; Gao et al., 2024). In a well-trained SAE, individual features in the hidden dimension align with the underlying sparse, semantically meaningful features.

## 2.2.1 Architecture and training

Sparse autoencoders (SAEs) are neural network models designed to learn compact, interpretable representations of high-dimensional data while enforcing sparsity in the hidden layer activations. The architecture of an SAE consists of an encoder network that maps the input to a hidden representation, and a decoder network that reconstructs the input from this representation.

Let x ∈ R d be an input vector, and h ∈ R n be the hidden representation, where typically n ≫ d . The encoder and decoder functions are defined as:

$$\text{Encoder} \, \colon \ \ h = f _ { \theta } ( x ) = \sigma ( W _ { e } x + b _ { e } )$$

$$\text{Decoder} \, \colon \quad \hat { x } = g _ { \phi } ( \mathbf h ) = W _ { d } \mathbf h + \mathbf b _ { d }$$

where W e ∈ R n × d and W d ∈ R d × n are the encoding and decoding weight matrices, b e ∈ R k and b d ∈ R d are bias vectors, and σ ( ) · is a non-linear activation function (e.g., ReLU or sigmoid). The parameters θ = { W , e b e } and ϕ = { W , d b d } are learned during training.

The training objective of our SAE combines three main components: a reconstruction loss, a sparsity constraint, and an auxiliary loss. The overall loss function is given by:

$$\mathcal { L } ( \theta, \phi ) = \frac { 1 } { d } \| { \mathbf x } - \hat { \mathbf x } \| _ { 2 } ^ { 2 } + \lambda \mathcal { L } _ { \text{sparse} } ( { \mathbf h } ) + \alpha \mathcal { L } _ { \text{aux} } ( { \mathbf x }, \hat { \mathbf x } )$$

where λ &gt; 0 and α &gt; 0 are hyperparameters controlling the trade-off between reconstruction fidelity, sparsity, and the auxiliary loss.

For the sparsity constraint, we use a k -sparse constraint: only the k largest activations in h are retained, while the rest are set to zero (Makhzani et al., 2013; Gao et al., 2024). This approach avoids issues such as shrinkage, where L1 regularisation can cause feature activations to be systematically lower than their true values, potentially leading to suboptimal representations shrinkage , (Wright et al., 2024; Rajamanoharan et al., 2024).

We also use an auxiliary loss, similar to the 'ghost grads' technique (Jermyn et al., 2023), to model the reconstruction error using the top k aux dead latents, where we typically set k aux = 2 k . Latents are flagged as dead during training if they have not activated for a predetermined number of tokens (in our case, one full epoch through the training data). Given the reconstruction error of the main model e = x -ˆ x , we define the auxiliary loss as:

$$\mathcal { L } _ { \text{aux} } ( \mathbf x, \hat { \mathbf x } ) = \| \mathbf e - \hat { \mathbf e } \| _ { 2 } ^ { 2 }$$

where ˆ = e W d z is the reconstruction using the top k aux dead latents, and z is the sparse representation using only these dead latents. This additional loss term helps to revive dead features and improve the overall representational capacity of the model (Gao et al., 2024).

## 2.2.2 Autointerpretability

As SAEs provide a framework for more interpretable representations, researchers have sought ways of automatically interpreting the features that they learn ( autointerpretability ). Gurnee et al. (2023) introduced sparse probing, where sparse linear models are used to detect the presence of concepts and features in internal model activations through supervised training. Many now use LLMs to directly interpret artifacts of unsupervised models i.e. SAEs. Bills et al. (2023) used GPT-4 to generate and simulate neuron explanations by looking at text on which the neuron activates strongly, while Bricken et al. (2023) and Templeton (2024) applied similar techniques to analyse sparse autoencoder features. Templeton (2024) further introduced a specificity analysis to rate explanations by using another LLM to predict activations based on the LLM-generated interpretation, providing a quantification of how interpretable a given neuron or feature actually is. Gao et al. (2024) demonstrated that cheaper methods, such as Neuron to Graph (Foote et al., 2023) with n -gram based explanations, allow for a scaleable feature labelling mechanism that does not rely on using more expensive LLMs.

## 2.2.3 Structure in SAE features

State-of-the-art automated interpretability techniques, as described in Section 2.2.2, have resulted in the discovery of a large volume of highly interpretable, monosemantic features in SAEs trained over language models (Cunningham et al., 2023a; Bricken et al., 2023). With features being the base unit of interpretability for SAEs, recent work has focused on understanding the geometric structure of features. Bricken et al. (2023) report feature splitting in geometrically close groups of semantically related features, where number of learned features in the cluster increases with model size. They also report the existence of universal features which re-occur between independent SAEs and which have

highly similar activation patterns. Templeton (2024) find feature splitting also occurs in SAEs trained over production-scale models, with larger SAEs also exhibiting novel features for concepts that are not represented in smaller SAEs. Makelov et al., 2024 report over-splitting of binary features. Engels et al., 2024 find clusters of SAE features that represent inherently multi-dimensional, non-linear subspaces.

## 3 Training SAEs and automated labelling

## 3.1 Training and automated interpretability methods

Training: We train our topk SAEs on the embeddings of abstracts from papers on arXiv with the astro-ph tag (astrophysics, 272,000 papers) and the cs.LG tag (computer science, 153,000 papers). The embeddings were generated with OpenAI's text-embedding-3-small model. 2 We train our SAEs on these collections of embeddings separately. We normalised the embeddings to zero mean and unit variance before passing them to the SAE as inputs. Our trained SAEs are available for download here.

Hyperparameters: Notable hyperparameters include the number of active latents k , the total number of latents n , the number of auxiliary latents k aux , the learning rate, and the auxiliary loss coefficient α . We found learning rate and auxiliary loss coefficient to not have a significant effect on final reconstruction loss; we set the former to 1e-4 and the latter to 1/32. We vary k between 16 and 128, and n between two to nine times the embedding dimension d input . Whilst we train SAEs with many different combinations of these hyperparameters, we largely focus on what we hereon refer to as SAE16 ( k = 16 , n = 2 d input = 3072 ), SAE32 ( k = 32 , n = 4 d input = 6144 ) and SAE64 ( k = 64 , n = 6 d input = 9216 ). We train each model for approximately 13.2 thousand steps.

Automated interpretability: Following the training of a Sparse Autoencoder (SAE), it becomes necessary to interpret its features, each corresponding to a column in the learned decoder weight matrix. To facilitate feature interpretation and quantify interpretation confidence, we employ two Large Language Model (LLM) instances: the Interpreter and the Predictor . The Interpreter is tasked with generating labels for each feature. It is provided with the abstracts that produce the top 5 activations of the feature across the dataset, along with randomly selected abstracts that do not activate the feature. The Interpreter then generates a label for the feature based on this input (for the complete prompt, refer to Appendix C). Subsequently, the generated label is passed to the Predictor. The Predictor is presented with three randomly sampled abstracts where the feature was activated and three where it was not. It is then instructed to predict whether a given abstract should activate the feature, expressing its confidence as a score ranging from -1 (absolute certainty of non-activation) to +1 (absolute certainty of activation). 3 We measure the Pearson correlation between this confidence and the true activation (binary; +1 or -1). We also measure the F1 score, when framing the confidence as a binary classification (active if confidence is above 0, inactive otherwise).

Evaluation metrics: In order to compare SAEs, we evaluate both their ability to reconstruct the embeddings, as well as the interpretability of the learned features. For the former, we examine the normalised mean squared error (MSE), where we divide MSE by the error when predicting the mean activations. We also report the log density of the activation of features across all papers. We do not report dead latents (those not firing on any abstract) as all models contained zero dead latents at the end of training. We also report the mean activation of features, when their activation is non-zero. To measure interpretability, we use Pearson correlation, as outlined above.

## 3.2 SAE Performance

We observe precise power-law scalings for sparse autoencoder (SAE) performance as a function of the number of total latents n , active latents k , and compute C used for training. The normalised mean squared error (MSE) scales as L n ( ) = cn -α for fixed k , where α ranges from 0.12 to 0.18, increasing with k , while c generally decreases (Figure 2, left panel). The cs.LG dataset shows slightly higher α values compared to astro-ph . For compute scaling, we calculate the number of training FLOPs

2 https://openai.com/index/new-embedding-models-and-api-updates/

3 We use 3 activating and 3 non-activating abstracts for the Predictor, rather than 5, due to LLM costs. We used gpt-4o as the Interpreter and gpt-4o-mini as the Predictor. Notably, we predict each abstract separately, rather than batching abstracts like Bricken et al. (2023).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-62242c9d-7190-4439-ac21-71923385c937.jpg)

Figure 2: Scaling laws for sparse autoencoder performance. Left: Normalised mean squared error (MSE) as a function of the number of total latents n for different values of active latents k . The power-law scaling is evident for each k . Right: Reconstruction loss as a function of compute (FLOPs) for different k values, demonstrating the compute-optimal model size scaling.

Figure 3: Pearson correlations between the ground-truth and predicted feature activation, using GPT-4o as the Interpreter and GPT-4o-mini as the Predictor .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-abe6c681-862a-40f0-bb81-ada843cb7c8b.jpg)

Figure 4: Sample feature family from cs.LG ; arrows represent C norm ij &gt; . 0 1 , with size ∝ C norm ij .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-7305d3e3-7b9b-4d5d-8d94-7c3231abebf6.jpg)

C at each step for each SAE. We find L C ( ) = aC b , where a generally increases with k (3.84 for k = 16 to 8.03 for k = 64 ) and b ranges from -0.11 to -0.16, becoming more negative as k increases from 16 to 64 (Figure 2, right panel). Both relationships show high accuracy with R-squared values above 0.93. Detailed fits are provided in Appendix A.

## 3.3 Interpretability of SAE features

The most direct way to evaluate the interpretability of features is to look at the distribution of automated interpretability scores, discussed above. Specifically: given a feature label from our interpreter model, how well can a predictor model predict the feature's activation on unseen text? We show in Figure 3 that the Pearson correlation between predictor model confidence of a feature firing and the ground-truth firing is quite high, with median correlations ranging from 0.65 to 0.71 for cs.LG and 0.85 to 0.98 for astro-ph . We note that Pearson correlation increases as k and n decrease, likely due to models learning coarser-grained features that are easier for the interpreter to identify.

## 4 Constructing feature families through graph-based clustering

We find that our SAEs trained over arXiv paper embeddings recover a wide range of features. These features cover both scientific concepts, from niche to broad and multi-disciplinary, and also more

abstract semantic artifacts, such as humorous writing or critiques of scientific theories. Features and activating examples can be found in Appendix C.

To analyse how features evolve across different SAE capacities and to identify meaningful groupings of related features, we studied two distinct phenomena: feature splitting and feature families . Feature splitting - the tendency of features appearing in larger SAEs to 'split' the direction spanned by a feature from a smaller SAE, and activate on granular sub-topics of the smaller SAE's feature - has been observed in previous work on sparse autoencoders (e.g. Bricken et al., 2023). Examples of feature splitting, as well as features recurring across SAEs, can be found in Figures 16 and 17a/17b.

In contrast, feature families exist within a single SAE, and exhibit a clear hierarchical structure with a dense 'parent' feature and several sparser 'child' features; we suggest that the 'parent' feature encompasses a broader, more abstract concept that is shared among the 'child' features. An example feature family from cs.LG can be seen in Figure 4.

## 4.1 Feature splitting

We investigated how features in smaller SAEs relate to features in larger SAEs through a nearest neighbour approach. For each pair of SAEs (i.e. SAE16 and SAE32) with n 1 and n 2 features respectively, we calculated an n 1 × n 2 similarity matrix S where S ij = w w T i j / ∥ w i ∥∥ w j ∥ . Here, w i and w j are decoder weight vectors for features in the smaller and larger SAE, respectively. For each feature in the larger SAE, we identified the most similar feature in the smaller SAE, allowing us to trace how features potentially 'split' or become more refined as model capacity increases.

Our results are shown in Figure 9. We find that increasing both number of active latents k and the latent dimension n reduces the similarity between nearest neighbours in differently sized SAEs. This agrees with intuition: larger models with more capacity (higher k and n ) can learn more fine-grained and specialised features, leading to greater differentiation from features in smaller models.

Qualitatively, matching features from small to large SAEs, we find both recurrent features and novel features. Recurrent features appear with extremely high S ij and activation similarity across one or more model pairs, and have highly similar auto-generated interpretations, suggesting semantic closeness; these are much more common for lower k (see Figure 16). In contrast, novel features have distinct semantic meaning from their nearest-neighbour match, and activate similarly on some but not all documents; novel features thus split the semantic space covered by their nearest-neighbour match from a smaller SAE. However, some novel features share little semantic or activation overlap with their nearest-neighbour feature, as in Fig. 17b, indicating smaller SAEs may not sufficiently cover the feature space; see E.1 in the Appendix for more details.

## 4.2 Feature families

Feature family identification To identify feature families, we developed a graph-based approach using co-activation patterns across the dataset. We consider only highly interpretable features (F1 ≥ 0 8 . , Pearson ≥ 0 8 . ).

We first compute co-occurrence matrix C and activation similarity matrix D . For all data points k :

$$C _ { i j } & = \sum _ { k } A _ { i k } A _ { j k } & D _ { i j } & = \sum _ { k } B _ { i k } B _ { j k }$$

$$i _ { j k }$$

where A ik = 1 if feature i is active on example k (0 otherwise), and B ik = h k i , if feature i is active on example k with hidden vector h k (0 otherwise). We normalise the co-occurrence matrix by feature activation frequencies and apply a threshold to focus on significant relationships:

$$C _ { i j } ^ { n o r m } = \frac { C _ { i j } } { f _ { i } + \epsilon }$$

where f i = ∑ k A ik is the activation frequency of feature i and ϵ is a small constant for numerical stability. We then apply a threshold τ to obtain C thresh ij (hereafter just C ). We construct a maximum spanning tree (MST) from C , capturing the strongest relationships between features while avoiding cycles. We convert the MST to a directed graph, with edges pointing from higher-density to lowerdensity features, representing a hierarchy from more general to more specific concepts. We identify feature families via depth-first-search in this directed graph, starting from root nodes (i.e., no incoming edges) and recursively exploring hierarchical sub-families.

We iterate this process, removing parent features after each iteration to re-form the MST and reveal overlapping, finer-grained feature families. We de-duplicate families with high set overlap ( | F 1 ∩ F 2 | | F 1 ∪ F 2 | &gt; . 0 6 ). In practice, we choose τ = 0 1 . and use n = 3 iterations.

Feature family interpretability To evaluate the interpretability of feature families, we analysed their collective properties and the effectiveness of high-level descriptions in capturing their behaviour. For each family, we generated a 'superfeature' description using GPT-4o, based on the individual feature descriptions within that family. We then uniformly sampled high-activating examples across all activations of child features, and assessed the interpretability of the superfeature using a prediction task, where GPT-4o predicted whether test abstracts would activate the superfeature. We compared these predictions to ground truth activations to compute Pearson correlation and F1 scores. Additionally, we calculated several metrics to characterise the structure and coherence of the feature families. Table 1 presents the mean values of these metrics across all families for both the astro-ph and cs.LG datasets.

Matrix structure We conjecture that feature families are equivalent to diagonal blocks in some permutation of the co-occurrence matrix C and activation similarity matrix D . If feature families are indeed meaningful clusters in the graph, then in C and D in-block elements should co-activate much more strongly than off-diagonal elements. We also argue that due to the hierarchical nature of feature families, matrix 'blocks' are highly sparse, since child features all co-occur with the parent feature but rarely co-occur with one another. Subsets of the co-occurrence matrix, permuted by feature family, are shown in 5.

̸

Motivated by these structures, we compute the parent-child co-occurrence ratio R p, ( C ) for every family with parent feature p and children C , avg ( ∑ i ∈C A ip ) avg ( ∑ i ∈C ∑ j ∈C ,j = i A ij ) . We also permute the co-occurrence and activation similarity matrices by greedily selecting feature families, and compute the in-block to off-diagonal ratios C diag /C off and D diag /D off (excluding the i = j diagonal), capturing the clustering strength of the block diagonal. Statistics are listed in Table 1.

| Dataset   | ( k , n )   |   Size |   F1 |   Pearson |   R ( p, C ) |   C diag /C off |   D diag /D off |   f inc |
|-----------|-------------|--------|------|-----------|--------------|-----------------|-----------------|---------|
| astro-ph  | (16, 3072)  |      6 | 0.86 |      0.76 |        10.99 |            5.13 |            5.47 |    0.36 |
|           | (32, 6144)  |      6 | 0.86 |      0.73 |        11.75 |            4.72 |            5.87 |    0.31 |
|           | (64, 9216)  |      7 | 0.8  |      0.7  |         6.87 |            2    |            3.05 |    0.24 |
| cs.LG     | (16, 3072)  |      5 | 0.73 |      0.6  |         2.44 |            8.35 |            0.89 |    0.23 |
| cs.LG     | (32, 6144)  |      5 | 0.73 |      0.59 |         3.5  |            7.33 |            1.07 |    0.3  |
| cs.LG     | (64, 9216)  |      7 | 0.8  |      0.71 |         1.22 |            1.78 |            2.57 |    0.41 |

Table 1: Interpretability and structure metrics for feature families from astro-ph and cs.LG ; we report medians unless otherwise noted. f inc refers to the fraction of features belonging to a clean (F1 ≥ 0 8 . , Pearson ≥ 0 8 . ) feature family.

## 5 Evaluating effectiveness of search interventions

## 5.1 Intervening on embeddings with SAE features

As an implementation detail, we note that intervening on a feature by up- or down-weighting its hidden representation and then decoding is equivalent to directly adding the scaled feature vector to the final embedding. We explore an alternative process in Appendix G where we iteratively optimise the encoded decoded latents to minimise the difference between the desired feature activations and the actual activations.

## 5.2 Experiments

We incorporate SAE-based embedding interventions into a literature retrieval system for cs.LG and astro-ph . To assess the effectiveness of SAE feature intervention on semantic search, we evaluate the specificity and interpretability of feature-centric query modifications. We select random samples ( N = 50 each) real literature retrieval queries relevant to machine learning and astronomy, which are

Figure 5: Co-occurrence matrix C organised by a subset of 5 feature families each. Features in families are ordered by firing density, and the right-most feature is the parent. The un-filled block structure reflects the hierarchical nature of the feature family: all children co-occur with the parent, but few children fire with each other. Visually, this supports our clustering approach.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-6ae9fb8a-3f4c-4037-947d-6d271f778832.jpg)

answerable with information in papers from cs.LG and astro-ph . For each query, we return the top k = 10 most relevant papers using embedding cosine similarity, making up the original retrieval results R . We then select a random feature i in the topk from the query's hidden representation h q , and another orthogonal feature j that has no overlap with the topk ; we limit our selection only to features that are highly interpretable (F1 &gt; 0 9 . , Pearson &gt; 0 9 . ). Given these features, we create a modified query embedding with h ′ q i , = λ -and h ′ q j , = λ + , letting λ -= 0 and sampling λ + ∈ [0 , 5] . This effectively 'down-weights' and 'up-weights' the importance of i and j , respectively, in the modified query, which is used to generate new retrieval results R ′ .

To the effect of up-weighting and down-weighting query modifications on the end retrieval results, we provide both R and R ′ to an external LLM instance. The external LLM then compares R and R ′ and determines which features, out of a multiple-choice subset of 5 options, have been up-weighted or down-weighted; we use this to compute the intervention accuracy, which measures the precision and efficacy of causal query interventions. As a baseline, we compare our SAE-based method against traditional query rewriting, by using another LLM instance to re-write the original query such that it up-weights j and down-weights i entirely using natural language.

Our results are shown in Figure 6. We find that SAE feature interventions consistently outperform traditional query rewriting across various levels of query fidelity. This Pareto improvement demonstrates that our method can achieve higher intervention accuracy while maintaining greater similarity to the original query. For instance, at a cosine similarity of 0.75, SAE interventions achieve approximately 20% higher accuracy compared to query rewriting.

We also experiment with intervening on feature families, sampling highly interpretable families containing features in the query topk . We uniformly adjust weights for all features in the family, including the parent, using the auto-generated family interpretation as the multiple-choice option. Results show that feature family interventions achieve accuracy comparable to individual features, but only down-weighting interventions outperform query re-writing. This may be because feature families can comprehensively down-weight related concepts, while up-weighting a general concept doesn't necessarily require activating all granular child features. Notably, lower cosine similarity isn't

Figure 6: Relationship between intervention accuracy and query fidelity for SAE-based embedding interventions versus traditional query rewriting in literature retrieval for cs.LG and astro-ph domains. Intervention accuracy measures the precision of causal query modifications, while query fidelity is quantified by cosine similarity between original and modified query embeddings.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-2cd9929c-3e18-441f-b0bc-127b5402867d.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-5ffd1fed-8378-4bb3-8313-08f39dd96e2b.jpg)

inherently undesirable, as changing the query will naturally reduce similarity. These experiments simply demonstrate that SAE features provide a way to parameterise semantic aspects of a query, allowing controlled modifications of its meaning.

## 6 Discussion

In this work, we have presented the first application of sparse autoencoders (SAEs) to dense text embeddings derived from large language models. By training SAEs on embeddings of scientific paper abstracts, we have demonstrated their effectiveness in disentangling semantic concepts while maintaining semantic fidelity. We introduced the concept of 'feature families' in SAEs, which allow for multi-scale semantic analysis and manipulation. Furthermore, we showcased the practical utility of our approach by applying these interpretable features to enable fine-grained control over query semantics. This aligns with recent work on controlled text generation, where Lee (2024) proposed using SAE-based interventions on embeddings to precisely alter generated text in specific semantic directions.

Importantly, our approach offers a novel solution to a growing challenge in scientific literature exploration. With the exponential proliferation of research papers, traditional methods of browsing and discovering relevant literature are becoming increasingly ineffective (Tsang et al., 2016). Our SAE-based method provides a new way to navigate and find pertinent papers, especially when researchers are not entirely certain about what they're looking for-a key function that libraries used to serve (Dahl, 2021). By allowing users to explore and manipulate interpretable semantic features, our system enables a more intuitive search process. This capability is particularly valuable in interdisciplinary research or when exploring emerging fields, where relevant work may not be easily discoverable through conventional keyword searches or citation networks (Sharma et al., 2022; Thomsett-Scott et al., 2016).

By providing a proof-of-concept for extracting interpretable features from dense embeddings, our work opens several promising research directions and applications across various NLP tasks. In text classification, our method could offer fine-grained insights into model decision boundaries with global features. For machine translation, it could enable targeted semantic manipulations, potentially addressing issues like gender bias in translations (Stanovsky et al., 2019). Scaling this approach to larger and more diverse datasets could reveal insights into how feature families and interpretability evolve with data size and domain breadth. While our current SAEs are trained on narrow scientific domains, extending this to the entirety of arXiv or even internet-scale text corpora could yield generalpurpose SAEs with exceptionally rich feature spaces, providing insights into the structure of human knowledge as represented in large-scale text embeddings (Thompson et al., 2020). Beyond these applications, our work contributes to the broader goal of making language models more transparent and controllable, which is crucial for building trust in AI systems as they become more integrated into critical decision-making processes (Doshi-Velez et al., 2017).

On an evaluation note, we'd like to see how our reconstructed embeddings fare on a standard semantic embedding benchmark such as MTEB (Muennighoff et al., 2022) in comparison to the original

underlying embeddings. We'd also like to be able to conduct an evaluation of SAE features against some proxy of ground-truth features, much like Makelov et al. (2024) propose. For instance, the Unified Astronomy Thesaurus (Frey et al., 2018) could provide a basis for evaluating individual feature overlap with astronomy concepts, and even family features as groupings of these individual concepts.

In regard to scientific discovery, our method offers a powerful tool for analysing the evolution of scientific fields over time. By examining statistics of SAE features-such as clustering patterns, co-occurrences, and temporal trends-researchers could gain novel insights into how scientific domains have changed and interacted. This approach could significantly enhance existing efforts to map the landscape of scientific research, which have relied primarily on citation networks and keyword analysis (Boyack et al., 2005). For instance, in astronomy, Sun et al. (2024) have used knowledge graphs to track field evolution; our SAE-based method could provide a more nuanced and interpretable view of conceptual shifts within the field. Furthermore, the interpretability offered by our SAEs could facilitate interdisciplinary research by providing a common semantic framework for comparing concepts across different scientific domains (Uzzi et al., 2013).

## 6.1 Limitations

Our work focused on relatively small datasets from specific scientific domains. Although this specificity allowed us to demonstrate the effectiveness of our approach in targeted areas, future work should investigate how well these methods generalise to larger, more diverse datasets. Additionally, our automated interpretability process, while effective, does not utilise the full spectrum of activations, potentially missing nuanced patterns in feature behaviour.

The computational requirements for training SAEs on large embedding datasets also present scalability challenges that need to be addressed for wider adoption of this approach. Our SAEs are quite small in comparison to more general language model SAEs. This proved adequate given the narrow domains we trained on, but SAEs for general text embeddings would need to be scaled up by at least 2-3 the total number of latents.

Acknowledgments. Part of this work was done at the 2024 Jelinek Memorial Summer Workshop on Speech and Language Technologies and was supported with discretionary funds from Johns Hopkins University and from the EU Horizons 2020 program's Marie Sklodowska-Curie Grant No 101007666 (ESPERANTO). Advanced Research Computing at Hopkins provided cloud computing to support the research. We are also grateful for the support from OpenAI through the OpenAI Researcher Access Program. Christine Ye is supported by the Regeneron Science Talent Search. Charles O'Neill is supported by a Tuckwell Scholarship at the Australian National University. Support for KI was provided by NASA through the NASA Hubble Fellowship grant HST-HF2-51508 awarded by the Space Telescope Science Institute, which is operated by the Association of Universities for Research in Astronomy, Inc., for NASA, under contract NAS5-26555.

## References

Baeza-Yates, Ricardo, Berthier Ribeiro-Neto, et al. (1999). Modern information retrieval . Vol. 463. ACM press New York.

Bills, Steven, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders (2023). 'Language models can explain neurons in language models'. In: URL https://openaipublic. blob. core. windows. net/neuronexplainer/paper/index. html.(Date accessed: 14.05. 2023) 2.

Boyack, Kevin W, Richard Klavans, and Katy B"orner (2005). 'Mapping the backbone of science'. In: Scientometrics 64, pp. 351-374.

Bricken, Trenton, Catherine Olsson, and Neel Nanda (2023). 'Towards Monosemanticity: Decomposing Language Models With Dictionary Learning'. In: arXiv preprint arXiv:2301.05498 .

Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. (2020). 'Language Models are Few-Shot Learners'. In: Advances in neural information processing systems 33, pp. 1877-1901.

Cao, Wenqiang, Qing Li, Siying Zhang, Rixin Xu, and Youqi Li (2023a). 'STEP: Generating Semantic Text Embeddings with Prompt'. In: 2023 Eleventh International Conference on Advanced Cloud

- and Big Data (CBD) , pp. 180-185. URL: https://api.semanticscholar.org/CorpusID: 269628678 .
- Cao, Yichen, Xinyi Wang, Yiran Cao, Renfeng Xu, Zhihan Dong, Qi Fang, Yeyun Gong, Lei Li, Shuming Shi, Jiafeng Yan, et al. (2023b). 'Step-gs: Guiding large language models via step-by-step prompting'. In: arXiv preprint arXiv:2305.11725 .
- Conmy, Arthur and Neel Nanda (2024). Activation Steering with SAEs . Accessed 16-07-2024. URL: https://www.lesswrong.com/posts/C5KAZQib3bzzpeyrg/full- post- progressupdate-1-from-the-gdm-mech-interp-team#Activation\_Steering\_with\_SAEs .
- Cunningham, Hoagy, Aidan Ewart, Logan Riggs, Robert Huben, and Lee Sharkey (2023a). Sparse Autoencoders Find Highly Interpretable Features in Language Models . arXiv: 2309.08600 [cs.LG] . URL: https://arxiv.org/abs/2309.08600 .
- -(2023b). 'Sparse autoencoders find highly interpretable features in language models'. In: arXiv preprint arXiv:2309.08600 .
- Dahl, Mark (2021). 'The evolving role of library collections in the broader information ecosystem'. In: Future Directions in Digital Information . Elsevier, pp. 161-174.
- Daujotas, Gytis (2024). Interpreting and Steering Features in Images . https://www.lesswrong. com/posts/Quqekpvx8BGMMcaem/interpreting-and-steering-features-in-images . [Accessed 16-07-2024].
- Deerwester, Scott, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman (1990). 'Indexing by latent semantic analysis'. In: Journal of the American society for information science 41.6, pp. 391-407.
- Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2018). 'BERT: Pretraining of Deep Bidirectional Transformers for Language Understanding'. In: arXiv preprint arXiv:1810.04805 .
- Donoho, David L (2006). 'Compressed sensing'. In: IEEE Transactions on Information Theory 52.4, pp. 1289-1306.
- Doshi-Velez, Finale and Been Kim (2017). Towards A Rigorous Science of Interpretable Machine Learning . arXiv: 1702.08608 [stat.ML] . URL: https://arxiv.org/abs/1702.08608 .
- Elhage, Nelson, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah (2022a). Toy Models of Superposition . arXiv: 2209.10652 [cs.LG] . URL: https://arxiv.org/abs/2209.10652 .
- Elhage, Nelson, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Johnston, Ben Mann, Amanda Askell, Danny Hernandez, Dawn Drain, Zac Hatfield-Dodds, et al. (2022b). 'Softmax Linear Units'. In.
- Engels, Joshua, Isaac Liao, Eric J. Michaud, Wes Gurnee, and Max Tegmark (2024). Not All Language Model Features Are Linear . arXiv: 2405.14860 [cs.LG] . URL: https://arxiv.org/abs/ 2405.14860 .
- Foote, Alex, Neel Nanda, Esben Kran, Ioannis Konstas, Shay Cohen, and Fazl Barez (2023). 'Neuron to graph: Interpreting language model neurons at scale'. In: arXiv preprint arXiv:2305.19911 .
- Frey, Katie and Alberto Accomazzi (2018). 'The Unified Astronomy Thesaurus: Semantic metadata
- for astronomy and astrophysics'. In: The Astrophysical Journal Supplement Series 236.1, p. 24. Furnas, George W, Thomas K Landauer, Louis M Gomez, and Susan T Dumais (1987). 'The vocabulary problem in human-system communication'. In: Communications of the ACM 30.11, pp. 964-971.
- Gao, Leo, John Thickstun, Anirudh Madaan, Zach Scherlis, Arush Guha, Sumanth Dathathri, Jared Kaplan, Azalia Mirhoseini, and Ilya Sutskever (2024). 'Scaling Laws for Neurons in GPT Models'. In: arXiv preprint arXiv:2401.02325 .
- Gao, Luyu, Xueguang Ma, Jimmy Lin, and Jamie Callan (2022). Precise Zero-Shot Dense Retrieval without Relevance Labels . arXiv: 2212.10496 [cs.IR] . URL: https://arxiv.org/abs/ 2212.10496 .
- Gao, Tianyu, Xingcheng Yao, and Danqi Chen (2021). 'SimCSE: Simple contrastive learning of sentence embeddings'. In: arXiv preprint arXiv:2104.08821 .
- Gurnee, Wes, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas (2023). 'Finding neurons in a haystack: Case studies with sparse probing'. In: arXiv preprint arXiv:2305.01610 .
- Jermyn, Adam and Adly Templeton (2023). Ghost Grads: An improvement on resampling . [Accessed 19-07-2024]. URL: https://transformer-circuits.pub/2024/jan-update/index. html#dict-learning-resampling .

Kingma, Diederik P and Jimmy Ba (2014). 'Adam: A method for stochastic optimization'. In: arXiv preprint arXiv:1412.6980 .

- Lee, Linus (2024). Prism: mapping interpretable concepts and features in a latent space of language . Accessed 16-07-2024. URL: https://thesephist.com/posts/prism .
- Liu, Nelson F, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith (2019). 'Linguistic knowledge and transferability of contextual representations'. In: arXiv preprint arXiv:1903.08855 .
- Makelov, Aleksandar, George Lange, and Neel Nanda (2024). 'Towards principled evaluations of sparse autoencoders for interpretability and control'. In: arXiv preprint arXiv:2405.08366 .
- Makhzani, Alireza and Brendan Frey (2013). 'K-sparse autoencoders'. In: arXiv preprint arXiv:1312.5663 .
- Manning, Christopher D, Prabhakar Raghavan, and Hinrich Schütze (2008). Introduction to information retrieval . Cambridge university press.
- Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean (2013a). 'Efficient estimation of word representations in vector space'. In: arXiv preprint arXiv:1301.3781 .
- Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean (2013b). 'Distributed representations of words and phrases and their compositionality'. In: Advances in neural information processing systems 26.
- Muennighoff, Niklas, Nouamane Tazi, Loic Magne, and Nils Reimers (2022). 'MTEB: Massive text embedding benchmark'. In: arXiv preprint arXiv:2210.07316 .
- Nanda, Neel (2023). Open Source Replication &amp; Commentary on Anthropic's Dictionary Learning Paper . [Accessed 22-07-2024]. URL: https://www.alignmentforum.org/posts/ fKuugaxt2XLTkASkk/open-source-replication-and-commentary-on-anthropic-s .
- Ng, Andrew et al. (2011). 'Sparse autoencoder'. In: CS294A Lecture notes . Vol. 72. 2011, pp. 1-19. Olshausen, Bruno A and David J Field (1997). 'Sparse coding with an overcomplete basis set: A strategy employed by V1?' In: Vision Research 37.23, pp. 3311-3325.
- Pennington, Jeffrey, Richard Socher, and Christopher D Manning (2014). 'Glove: Global vectors for word representation'. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pp. 1532-1543.
- Peters, Matthew E, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer (2018). 'Deep contextualized word representations'. In: arXiv preprint arXiv:1802.05365 .
- Rajamanoharan, Senthooran, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, and Neel Nanda (2024). 'Improving dictionary learning with gated sparse autoencoders'. In: arXiv preprint arXiv:2404.16014 .

Reimers, Nils, Lucas Beyer, and Iryna Wang (2022). 'The curse of dense low-dimensional information retrieval for large index sizes'. In: arXiv preprint arXiv:2112.07899 .

- Reimers, Nils and Iryna Gurevych (2019). 'Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks'. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , pp. 3982-3992.
- Sharma, Ritu, Sarita Gulati, Amanpreet Kaur, Atasi Sinhababu, and Rupak Chakravarty (2022). 'Research discovery and visualization using ResearchRabbit: A use case of AI in libraries'. In: COLLNET Journal of Scientometrics and Information Management 16.2, pp. 215-237.
- Stanovsky, Gabriel, Noah A Smith, and Luke Zettlemoyer (2019). 'Evaluating Gender Bias in Machine Translation'. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics .
- Sun, Zechang, Yuan-Sen Ting, Yaobo Liang, Nan Duan, Song Huang, and Zheng Cai (2024). Knowledge Graph in Astronomical Research with Large Language Models: Quantifying Driving Forces in Interdisciplinary Scientific Discovery . arXiv: 2406.01391 [astro-ph.IM] . URL: https: //arxiv.org/abs/2406.01391 .

Templeton, Adly (2024). Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet . Anthropic.

- Thompson, Nicholas, Kristjan Greenewald, Keegan Lee, and Gabriel Manso (2020). 'An analysis of cross-domain performance in deep neural networks'. In: arXiv preprint arXiv:2001.11264 .
- Thomsett-Scott, Beth and Patricia E Reese (2016). 'Academic libraries and discovery tools: A survey of the literature'. In: Discovery Tools: The Next Generation of Library Research , pp. 3-23.
- Trifonov, Valentin, Octavian-Eugen Ganea, Anna Potapenko, and Thomas Hofmann (2018). Learning and Evaluating Sparse Interpretable Sentence Embeddings . arXiv: 1809.08621 [cs.CL] . URL: https://arxiv.org/abs/1809.08621 .

Tsang, Daniel C and Julia M Gelfand (2016). 'The Changing Landscape of Research Library Collections: Ensuring Realistic Sustainability.' In.

- Turian, Joseph, Lev Ratinov, and Yoshua Bengio (2010). 'Word representations: a simple and general method for semi-supervised learning'. In: Proceedings of the 48th annual meeting of the association for computational linguistics , pp. 384-394.
- Uzzi, Brian, Satyam Mukherjee, Michael Stringer, and Ben Jones (2013). 'Atypical combinations and scientific impact'. In: Science 342.6157, pp. 468-472.
- Wang, Liang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei (2024). Text Embeddings by Weakly-Supervised Contrastive Pre-training . arXiv: 2212.03533 [cs.CL] . URL: https://arxiv.org/abs/2212.03533 .

Wright, Benjamin and Lee Sharkey (2024). Addressing Feature Suppression in SAEs . https://www. alignmentforum.org/posts/3JuSjTZyMzaSeTxKk/addressing-feature-suppressionin-saes . [Accessed 16-07-2024].

## Contents

1

Introduction

| 2   | 2                                  | Background                                              | and Related work                                        |   2 |
|-----|------------------------------------|---------------------------------------------------------|---------------------------------------------------------|-----|
|     | 2.1                                | Embeddings and Representation Learning .                | Embeddings and Representation Learning .                |   2 |
|     | 2.2                                | Sparse autoencoders .                                   | . . . . . .                                             |   3 |
|     |                                    | 2.2.1                                                   | Architecture and training                               |   3 |
|     |                                    | 2.2.2                                                   | Autointerpretability . . .                              |   4 |
|     |                                    | 2.2.3                                                   | Structure in SAE features                               |   4 |
| 3   | Training                           | SAEs                                                    | and automated labelling                                 |   5 |
|     | 3.1                                | Training and automated interpretability methods . . . . | Training and automated interpretability methods . . . . |   5 |
|     | 3.2                                | SAE Performance . . . . . . . .                         | SAE Performance . . . . . . . .                         |   5 |
|     | 3.3                                | Interpretability of SAE features . . . . . .            | Interpretability of SAE features . . . . . .            |   6 |
| 4   |                                    | Constructing                                            | feature families through                                |   6 |
|     | 4.1                                | Feature splitting . . . . . . .                         | . .                                                     |   7 |
|     | 4.2                                | Feature families . . . . .                              | . . . .                                                 |   7 |
| 5   |                                    | Evaluating                                              | effectiveness of search interventions                   |   8 |
|     | 5.1                                | Intervening on embeddings with SAE features . . . . .   | Intervening on embeddings with SAE features . . . . .   |   8 |
|     | 5.2                                | Experiments . . . . . . . . . .                         | .                                                       |   8 |
| 6   | Discussion                         | Discussion                                              | Discussion                                              |  10 |
|     | 6.1                                | Limitations . . . . . .                                 | . . . . .                                               |  11 |
| A   | Training details                   | Training details                                        | Training details                                        |  16 |
|     | A.1                                | Training setup . . . . . . . . . . .                    | Training setup . . . . . . . . . . .                    |  16 |
|     | A.2                                | SAE training metrics . . .                              | . . .                                                   |  16 |
|     | A.3                                | Scaling laws . . . .                                    | . . . . . . .                                           |  16 |
|     | A.4                                | Feature density and similarity .                        | .                                                       |  18 |
| B   | SAErch.ai                          | SAErch.ai                                               | SAErch.ai                                               |  18 |
|     | B.1                                | Overview . . . .                                        | . . . . . . . .                                         |  18 |
|     | B.2                                | Feature Visualisation Tab .                             | . . .                                                   |  19 |
|     |                                    | B.2.1                                                   | Individual Features . . .                               |  19 |
|     |                                    | B.2.2                                                   | Feature Families . . . .                                |  20 |
| C   | Automated interpretability details | Automated interpretability details                      | Automated interpretability details                      |  20 |
|     | C.1                                | Examples of features .                                  | . . . . .                                               |  20 |
|     | C.2                                | Automated interpretability prompts . . . . . . . . .    | Automated interpretability prompts . . . . . . . . .    |  22 |
|     | C.3                                | Exploring the effectiveness of smaller models           | Exploring the effectiveness of smaller models           |  24 |

1

| D Cross-domain features   | D Cross-domain features                    |   24 |
|---------------------------|--------------------------------------------|------|
| E Feature family details  | E Feature family details                   |   26 |
| E.1                       | Feature splitting structures . . . . . . . |   26 |
| E.2                       | Feature family structure . . . . . . . . . |   27 |
| E.3                       | Feature family interpretability . . . . .  |   30 |
| F                         | Exploring learned decoder weight matrices  |   30 |
| G                         | Iterative encoding optimisation            |   30 |

## A Training details

## A.1 Training setup

Our sparse autoencoder (SAE) implementation incorporates several recent advancements in the field. Following Bricken et al. (2023), we initialise the bias b pre using the geometric median of a data point sample and set encoder directions parallel to decoder directions. Decoder latent directions are normalised to unit length at initialisation and after each training step. For our topk models, based on Gao et al. (2024), we set initial encoder magnitudes to match input vector magnitudes, though our analyses indicate minimal impact from this choice.

We augment the primary loss with an auxiliary component (AuxK), inspired by the 'ghost grads' approach of Jermyn et al. (2023). This auxiliary term considers the topk aux inactive latents (typically k aux = 2 k ), where inactivity is determined by a lack of activation over a full training epoch. The total loss is formulated as L + α L aux , with α usually set to 1/32. This mechanism reduces the number of dead latents with minimal computational overhead (Gao et al., 2024). We found that dead latents only occurred during training the k = 16 models, and all dead latents had disappeared by the end of training. We show how dead latents evolved over training the k = 16 SAEs for the astro-ph abstracts in Figure 7.

For optimisation, we employ Adam (Kingma et al., 2014) with β 1 = 0 9 . and β 2 = 0 999 . , maintaining a constant learning rate. We use gradient clipping. Our training uses batches of 1024 abstracts, with performance metrics showing robustness to batch size variations under appropriate hyperparameter settings.

The primary MSE loss uses a global normalisation factor computed at training initiation, while the AuxK loss employs per-batch normalisation to adapt to evolving error distributions. Following Bricken et al. (2023), we apply a gradient projection technique to mitigate interactions between the Adam optimiser and decoder normalisation.

## A.2 SAE training metrics

Table 2 shows the final training metrics for all combinations of SAEs trained. We note clear trends in normalised MSE, log feature density and activation mean as we vary the number of active latents k and the overall number of latents n .

## A.3 Scaling laws

For the left panel of Figure 2, which shows the scaling of normalised MSE with the number of total latents n , we observe the following power-law relationships:

$$k$$

```
\begin{array} { c } w = 1 6 \, \colon L ( n ) = 0. 6 1 n ^ { - 0. 1 2 } \, ( a t r o. p h ) ; \, L ( n ) = 0. 6 7 n ^ { - 0. 1 3 } \, ( c s. L G ) \\ k = 3 2 \, \colon L ( n ) = 0. 4 9 n ^ { - 0. 1 3 } \, ( a t r o. p h ) ; \, L ( n ) = 0. 5 6 n ^ { - 0. 1 4 } \, ( c s. L G ) \\ k = 6 4 \, \colon L ( n ) = 0. 4 6 n ^ { - 0. 1 5 } \, ( a t r o. p h ) ; \, L ( n ) = 0. 6 0 n ^ { - 0. 1 7 } \, ( c s. L G ) \\ k = 1 2 8 \, \colon L ( n ) = 0. 3 1 n ^ { - 0. 1 3 } \, ( a t r o. p h ) ; \, L ( n ) = 0. 5 1 n ^ { - 0. 1 8 } \, ( c s. L G ) \end{array}
```

Figure 7: The proportion of dead latents, defined as features that haven't fired in the last epoch of training, for our k = 16 SAEs on the astro-ph abstract embeddings. All dead latents were gone by the end of training. We found that dead latents only occurred in k = 16 autoencoders.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-cd4ef864-ebb6-4798-8b2b-03162e7b4baa.jpg)

Table 2: Metrics for our topk sparse autoencoders with varying k and hidden dimensions, across both astronomy and computer science papers. MSE is normalised mean squared error, Log FD is the mean log density of feature activations, and activation mean is the mean activation value across non-zero features. Note that MSE is normalised.

|     |       | astro.ph   | astro.ph   | astro.ph   | cs.LG   | cs.LG   | cs.LG    |
|-----|-------|------------|------------|------------|---------|---------|----------|
| k   | n     | MSE        | Log FD     | Act Mean   | MSE     | Log FD  | Act Mean |
|     | 3072  | 0.2264     | -2.7204    | 0.1264     | 0.2284  | -2.7314 | 0.1332   |
|     | 4608  | 0.2246     | -4.7994    | 0.1350     | 0.2197  | -3.0221 | 0.1338   |
| 16  | 6144  | 0.2128     | -3.1962    | 0.1266     | 0.2089  | -3.2299 | 0.1342   |
|     | 9216  | 0.1984     | -3.4206    | 0.1264     | 0.1962  | -3.4833 | 0.1343   |
|     | 12288 | 0.1957     | -6.2719    | 0.1274     | 0.1897  | -3.6448 | 0.1347   |
|     | 3072  | 0.1816     | -2.3389    | 0.0847     | 0.1831  | -2.3008 | 0.0885   |
|     | 4608  | 0.1691     | -3.6091    | 0.0882     | 0.1697  | -2.5152 | 0.0876   |
| 32  | 6144  | 0.1604     | -2.7761    | 0.0841     | 0.1641  | -2.6687 | 0.0873   |
|     | 9216  | 0.1554     | -3.0227    | 0.0842     | 0.1540  | -2.9031 | 0.0875   |
|     | 12288 | 0.1520     | -4.9505    | 0.0843     | 0.1457  | -3.0577 | 0.0877   |
|     | 3072  | 0.1420     | -1.9538    | 0.0566     | 0.1485  | -1.8875 | 0.0584   |
|     | 4608  | 0.1331     | -2.7782    | 0.0622     | 0.1370  | -2.0637 | 0.0570   |
| 64  | 6144  | 0.1262     | -2.2828    | 0.0545     | 0.1310  | -2.1852 | 0.0558   |
|     | 9216  | 0.1182     | -2.4682    | 0.0539     | 0.1240  | -2.3536 | 0.0545   |
|     | 12288 | 0.1152     | -3.4787    | 0.0583     | 0.1162  | -2.4847 | 0.0548   |
|     | 3072  | 0.1111     | -1.8876    | 0.0483     | 0.1206  | -1.5311 | 0.0399   |
|     | 4608  | 0.1033     | -2.1392    | 0.0457     | 0.1137  | -1.6948 | 0.0376   |
| 128 | 6144  | 0.1048     | -2.2501    | 0.0438     | 0.1076  | -1.8079 | 0.0366   |
|     | 9216  | 0.0975     | -2.5352    | 0.0409     | 0.0999  | -1.9701 | 0.0348   |
|     | 12288 | 0.0936     | -2.7025    | 0.0399     | 0.0942  | -2.0858 | 0.0342   |

Figure 8: Log feature density for features in our three SAEs as a stacked histogram, showing the distribution of how often features fire across all paper abstacts ( cs.LG and astro-ph ). The larger SAE has a higher mean feature density than the smaller SAEs.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-4b872889-4f93-4e74-90f6-e14f4e559b40.jpg)

For the right panel of Figure 2, which shows the scaling of normalised MSE with the amount of compute C (in FLOPs), we observe the following power-law relationships:

$$\dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \\ k = 1 6 \colon L ( C ) = 3. 8 4 C ^ { - 0. 1 1 } \\ k = 3 2 \colon L ( C ) = 5. 2 5 C ^ { - 0. 1 3 } \\ k = 6 4 \colon L ( C ) = 8. 0 3 C ^ { - 0. 1 6 } \\ k = 1 2 8 \colon L ( C ) = 2. 8 0 C ^ { - 0. 1 3 } \\. \quad. \quad. \quad.. \quad..$$

These equations demonstrate the consistent power-law scaling behaviour of sparse autoencoders across different values of k , n , and compute C .

## A.4 Feature density and similarity

We find an intuitive relationship between k and n and the log feature density (essentially, how often a given feature fires). As k increases, we get a sharper peak of log feature density, shifted to the right, suggesting features fire in a tighter range as we increase the instantaneous L0 of the SAE's encoder (Figure 8).

To compare features across different SAEs trained on the same input data, we analyse the cosine similarity between the decoder weight vectors corresponding to each feature. Decoder weights, represented by columns in the decoder matrix, directly encode each feature's contribution to input reconstruction. Encoder weights, on the other hand, are optimised to extract feature coefficients while minimising interference between non-orthogonal features. This separation is important in the context of superposition, where we have more features than input dimensions, precluding perfect orthogonality.

## B SAErch.ai

To demonstrate the practical applications of our sparse autoencoder (SAE) approach to semantic search and feature interpretation, we developed SAErch.ai, a web application that allows users to interact with the SAE models trained on arXiv paper embeddings.

## B.1 Overview

SAErch.ai is built using the Gradio framework and consists of three main tabs: Home, SAErch, and Feature Visualisation. The application allows users to switch between the Computer Science ( cs.LG ) and Astrophysics ( astro-ph ) datasets.

The SAErch tab implements the core functionality of our semantic search system, allowing users to:

(a) k fixed, varying n . As n increases, the features between across SAEs with varying k become more disparate.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-6def377a-c879-47de-94ce-f67581acc9ae.jpg)

(b) n fixed, varying k . Higher values of k lead to less similarity regardless of n .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-dccde964-e84d-4216-a55e-2d1f38f9288b.jpg)

Figure 9: Nearest-neighbour cosine similarity distributions for SAE features. To find features in an SAE with a lower k that are most similar to those in an SAE with a larger k , we compute the cosine similarity between each feature in the larger model and each feature in the smaller model. We do this for several values of n , and combine the distributions for astro.ph and cs.LG .

- · Input a search query
- · View the top 10 search results based on embedding similarity
- · Interact with the SAE features activated by their query

For each query, the system displays sliders corresponding to the top-k SAE features activated by the input. Users can adjust these sliders to modify the query embedding, effectively steering the search results towards or away from specific semantic concepts; see Figure 10. This directly demonstrates the fine-grained control over query semantics discussed in Section 5 of our paper. Users can also search for and add specific features not initially activated by their query (Figure ?? ).

## B.2 Feature Visualisation Tab

The Feature Visualisation tab is divided into two sub-tabs: Individual Features and Feature Families. This section of the application directly relates to our analysis of SAE features and feature families discussed in Sections 3 and 4.

## B.2.1 Individual Features

For any selected feature, this tab displays:

- · Top 5 activating abstracts, demonstrating the semantic content captured by the feature
- · Top and bottom 5 correlated features, illustrating the relationships between different SAE features
- · Top 5 co-occurring features, showing which features tend to activate together
- · A histogram of activation values, providing insight into the feature's behavior across the corpus
- · The most similar features in SAE16 and SAE32

Figure 10: The SAErch tab of our web application, demonstrating a semantic search for 'measurable signatures of stochasticity in star formation in galaxies' in the astrophysics domain. The interface displays the top 10 search results ranked by relevance, including title, citation count, and publication year. On the right, sliders represent the top activated SAE features for the query, allowing users to fine-tune the search by adjusting feature weights. On the bottom we have our feature addition interface. Users can search for specific semantic features (e.g., 'black holes') and add them to their query. They can then adjust the strength of these features.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-721f79e6-13b4-43c3-b448-a10e6b8e4e19.jpg)

## B.2.2 Feature Families

The Feature Families tab in our web application offers an in-depth exploration of related features discovered by our sparse autoencoder. We show an example feature family in Figure 12.

The table displays the parent feature (superfeature) and its child features, along with key metrics, such as the name of the parent and child features, the frequency of co-occurrence between the child feature and the parent feature, ranging from 0 to 1, and the F1 Score and Pearson correlation.

The interactive directed graph provides a visual representation of the feature family structure. Each node represents a feature. The size of the node corresponds to the feature's density (frequency of activation), while the color intensity indicates the Pearson correlation (interpretability). Arrows between nodes show relationships between features, with the direction typically pointing from more general to more specific concepts. Users can hover over nodes to view detailed information about each feature, including its name and log density.

## C Automated interpretability details

## C.1 Examples of features

We show some examples of perfectly interpretable features (Pearson correlation &gt; . 0 99 ) in Table 3. The strength of the activation of the feature on its top 3 activating abstracts is shown in parentheses next to the abstract title.

| Circuit Discovery_with Differentiable Computation Graph Pruning                                                           |   0.2732 |
|---------------------------------------------------------------------------------------------------------------------------|----------|
| Dictionary_Learning_Inproves Patch-Eree Circuit Discovery in Mechanistic Interpretability: Case Study Othello-GPT         | 2362     |
| Compositional Atlas Of Iractable Circuit Operations: Eron_Simple Transformations to Complex Information-Theoretic Queries |    0.223 |
| Does_Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities_in Chinchilla                    | 2108     |
| to_investi regulatory_control circuits bacterial metabolic pathways 'gate                                                 | 2068     |

| Top 5 Correlated Features            |           | Bottom 5 Correlated Features             |
|--------------------------------------|-----------|------------------------------------------|
| in EDA for IC/VLSI optimization      | 0.316255  | Synthetic data quality and privacy       |
| Gating mechanisms in neural networks | 0.297054  | Hotel booking and recommendation systems |
| Novelty detection methodologies      | 296994    | High mobility communication optimization |
| Circular data and models             | 0.282449  | Advanced machine learning applications   |
| Deep learning for classification     | 0 .243455 | Adaptive algorithms (Ada-prefixed)       |

Figure 11: Individual feature visualisation for the 'Circuit analysis in neural networks' feature in the computer science domain. The interface displays key interpretability metrics, top activating abstracts, correlated and co-occurring features, and an activation distribution histogram. Further information (not shown in the image) includes co-occurring features and activation distribution.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-ce62f5d4-4437-48b6-b20a-ff405e30e249.jpg)

Figure 12: Directed graph visualization of a transformer models feature family. Nodes represent individual features, with size indicating feature density and color intensity showing Pearson correlation. Edges depict relationships between features, with arrow direction pointing from more general to more specific concepts. Users can hover over nodes to view detailed feature information.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-1340b647-0476-41fc-b236-e439aba2340d.jpg)

## Feature

Astronomy

Table 3: Activation strengths and titles for abstracts related to Astronomy and Computer Science features.

| Cosmic Microwave Background           | CMBmap-making and power spectrum estimation (0.1708)                                                                 | How to calculate the CMB spectrum (0.1598)                                                                    | CMB data analysis and spar- sity (0.1581)                                                                                   |
|---------------------------------------|----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| Periodicity in astronomical data      | Generalized Lomb-Scargle analysis of decay rate measurements from the Physikalisch-Technische Bundesanstalt (0.1027) | Multicomponent power- density spectra of Kepler AGNs, an instrumental artefact or a physical origin? (0.0806) | RXTE observation of the X- ray burster 1E 1724-3045. I. Timing study of the persistent X-ray emission with the PCA (0.0758) |
| X-ray reflection spectra              | X-ray reflection spectra from ionized slabs (0.3859)                                                                 | The role of the reflection fraction in constraining black hole spin (0.3803)                                  | Relativistic reflection: Re- view and recent develop- ments in modeling (0.3698)                                            |
| Critique or refutation of theories    | What if string theory has no de Sitter vacua? (0.2917)                                                               | No evidence of mass segrega- tion in massive young clusters (0.2051)                                          | Ruling Out Initially Clustered Primordial Black Holes as Dark Matter (0.2029)                                               |
| Computer Science                      |                                                                                                                      |                                                                                                               |                                                                                                                             |
| Sparsity in Neural Networks           | Two Sparsities Are Better Than One: Unlocking the Per- formance Benefits of Sparse- Sparse Networks (0.3807)         | Truly Sparse Neural Net- works at Scale (0.3714)                                                              | Topological Insights into Sparse Neural Networks (0.3689)                                                                   |
| Gibbs Sampling and Variants           | Herded Gibbs Sampling (0.2990)                                                                                       | Characterizing the General- ization Error of Gibbs Algo- rithm with Symmetrized KL information (0.2858)       | A Framework for Neural Net- work Pruning Using Gibbs Distributions (0.2843)                                                 |
| Arithmetic operations in transformers | Arbitrary-Length Generaliza- tion for Addition in a Tiny Transformer (0.1828)                                        | Carrying over algorithm in transformers (0.1803)                                                              | Understanding Addition in Transformers (0.1792)                                                                             |

## C.2 Automated interpretability prompts

We provide the prompts used for the Interpreter model and the Predictor model in the boxes below. Where this text is used, it represents an input to the model. We found that performance significantly increased when including the instruction to use 'Occam's razor', whereby the simplest feature at the appropriate level of granularity was selected.

## Interpreter Model Prompt

You are a meticulous &lt;type&gt; researcher conducting an important investigation into a certain neuron in a language model trained on &lt;subject&gt; papers. Your task is to figure out what sort of behaviour this neuron is responsible for - namely, on what general concepts, features, themes, methodologies or topics does this neuron fire? Here's how you'll complete the task:

INPUT DESCRIPTION : You will be given two inputs: 1) Max Activating Examples and 2) Zero Activating Examples.

- 1. You will be given several examples of text that activate the neuron, along with a number being how much it was activated. This means there is some feature, theme, methodology, topic or concept in this text that 'excites' this neuron.
- 2. You will also be given several examples of text that don't activate the neuron. This means the feature, topic or concept is not present in these texts.

OUTPUT DESCRIPTION: Given the inputs provided, complete the following tasks.

- 1. Based on the MAX ACTIVATING EXAMPLES provided, write down potential topics, concepts, themes, methodologies and features that they share in common. These will need to be specific - remember, all of the text comes from subject , so these need to be highly specific subject concepts. You may need to look at different levels of granularity (i.e. subsets of a more general topic). List as many as you can think of. Give higher weight to concepts more present/prominent in examples with higher activations.
- 2. Based on the zero activating examples, rule out any of the topics/concepts/features listed above that are in the zero-activating examples. Systematically go through your list above.
- 3. Based on the above two steps, perform a thorough analysis of which feature, concept or topic, at what level of granularity, is likely to activate this neuron. Use Occam's razor, as long as it fits the provided evidence. Be highly rational and analytical here.
- 4. Based on step 4, summarise this concept in 1-8 words, in the form FINAL: &lt;explanation&gt; . Do NOT return anything after these 1-8 words.

Here are the max-activating examples: &lt;max activating examples&gt;

Here are the zero-activating examples: &lt;zero activating examples&gt;

Work through the steps thoroughly and analytically to interpret our neuron.

## Predictor Model Prompt

You are a &lt;subject&gt; expert that is predicting which abstracts will activate a certain neuron in a language model trained on &lt;subject&gt; papers. Your task is to predict which of the following abstracts will activate the neuron the most. Here's how you'll complete the task:

INPUT DESCRIPTION: You will be given the description of the type of paper abstracts on which the neuron activates. This description will be short. You will then be given an abstract. Based on the concept of the abstract, you will predict whether the neuron will activate or not.

OUTPUT DESCRIPTION: Given the inputs provided, complete the following tasks.

- 1. Based on the description of the type of paper abstracts on which the neuron activates, reason step by step about whether the neuron will activate on this abstract or not. Be highly rational and analytical here. The abstract may not be clear cut - it may contain topics/concepts close to the neuron description, but not exact. In this case, reason thoroughly and use your best judgement. However, do not speculate on topics that are not present in the abstract.
- 2. Based on the above step, predict whether the neuron will activate on this abstract or not. If you predict it will activate, give a confidence score from 0 to 1 (i.e. 1 if you're certain it will activate because it contains topics/concepts that match the description exactly, 0 if you're highly uncertain). If you predict it will not activate, give a confidence score from -1 to 0.
- 3. Provide the final confidence score in the form PREDICTION: (your prediction) e.g. PREDICTION: 0.5 . Do NOT return anything after this.

Here is the description/interpretation of the type of paper abstracts on which the neuron activates: &lt;description&gt;

Here is the abstract to predict: &lt;abstract&gt;

Work through the steps thoroughly and analytically to predict whether the neuron will activate on this abstract.

## C.3 Exploring the effectiveness of smaller models

Although we eventually used gpt-4o-mini as the Predictor model, we initially did some ablations to understand how effective gpt-4o and gpt-3.5-turbo would be as different combinations of the Interpreter and Predictor models. We measured this by randomly sampling 50 features from our SAE64 (trained on astro-ph abstracts) and measuring the interpretability scores of different model combinations, in terms of both F1 score (does the model's binary classification of a feature firing on an abstract agree with the ground-truth) and the Pearson correlation (described in the main body). Interestingly, we observe that using gpt-4o as the Interpreter and gpt-3.5-turbo as the Predictor leads to similar scores as using gpt-3.5-turbo for both, as shown in Figures 13 and Figures 14. This suggests that the challenging task in the autointerp is not necessarily labelling but rather predicting the activation of a feature on unseen abstracts.

Another observation is that using gpt-3.5-turbo as the Predictor only leads to a moderate degradation of F1 score, it leads to a significant degradation of Pearson correlation. This is likely because we only use 6 abstracts for each feature prediction (3 positive, 3 negative) and thus there are only a few discrete F1 scores possible. Additionally, it appeared that gpt-3.5-turbo was generally less likely to assign higher confidence scores in either direction, with a much lower variance in assigned confidence than when gpt-4o was the Predictor. This affects Pearson correlation but not F1.

## D Cross-domain features

The intersection between our cs.LG ( n = 153 146 , ) and astro.PH ( n = 271 492 , ) corpora contains n = 330 cross-posted papers. Motivated by these papers, as well as the observation of similar features re-occurring in models of different sizes (see Section 4), we search for the max cosine similarity feature between cs.LG and astro.PH SAEs at a fixed k and n dir . As expected, we find

Figure 13: Correlation between F1 scores and Pearson correlation scores of different combinations of (labeller, predictor) models. Interestingly, using GPT-3.5 as the predictor appears to degrade performance similarly regardless of whether the feature was labelled by GPT-4o or GPT-3.5.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-01866706-09e2-45b9-851f-1acc595b9391.jpg)

Figure 14: Mean F1 scores and Pearson correlations (according to ground-truth feature activations) across 50 randomly sampled features, for different combinations of (Interpreter, Predictor) models.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-81703f0e-0da2-4cd8-aa04-4436a14d7be7.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-d276ee57-9add-470e-a272-efc23bc8e992.jpg)

Figure 15: Maximum pair-wise cosine similarity of feature vectors between SAEs trained on different domains.

| Feature Name ( astro-ph )       | Best Match ( cs.LG )                       |   Cosine Sim. |   Activation Sim. |   ∆ F1 |   ∆ Pearson |
|---------------------------------|--------------------------------------------|---------------|-------------------|--------|-------------|
| Deep learning                   | CNNs and Applications                      |          0.39 |              0.33 |  -0.2  |       -0.17 |
| Generative Adversarial Networks | Generative Adversarial Networks (GANs)     |          0.61 |              0.26 |   0    |        0    |
| Transformers                    | Transformer architectures and applications |          0.5  |              0.33 |   0    |       -0    |
| Artificial Neural Networks      | Artificial Neural Networks (ANNs)          |          0.64 |              0.02 |   0    |        0    |
| Artificial Intelligence         | AI applications in diverse domains         |          0.61 |              0.45 |   0    |        0.02 |
| Automation and Machine Learning | Automation in computational processes      |          0.9  |              0.77 |  -0.25 |       -0.47 |
| Gaussian Processes              | Gaussian Processes in Machine Learning     |          0.59 |              0.54 |   0    |        0.03 |
| Regression analysis             | Regression techniques and applications     |          0.81 |              0.53 |   0    |       -0.01 |

Table 4: Feature matches from the "Machine Learning" family ( astroPH ); k = 64 , n dir = 9216 .

significant mis-alignment between the vast majority of feature vectors between SAEs trained on different domains, with mis-alignment increasing with k and n dir (see Figure 15; this is unsurprising given how k and n dirs correlate with feature granularity).

However, a small subset of features appear in both sets of SAEs, with relatively high max cosine similarity. For example, Table 4 shows the nearest cs.LG neighbours for every feature in the astro.PH 'Machine Learning' feature family (average cosine similarity = 0 59 . , average activation similarity = 0 40 . ). To test whether the features represent the same semantic concepts, we substitute the natural language description of the best-match cs.LG feature for each listed astro.PH feature and test the interpretability of the substituted descriptions; we find ∆ Pearson = -0 07 . and ∆ F 1 = -0 06 . . The existence of these features suggests that both sets of SAEs learn a semi-universal set of features that span the domain overlap between astro.PH and cs.LG .

Interestingly, we find a number of near-perfectly aligned pairs (cosine similarity &gt; 0 95 . ) of highly interpretable features with little semantic overlap. A number of these features share similar wording but not meaning, such as 'Substructure in dark matter and galaxies" ( astro-ph ) and 'Subgraphs and their representations". Of these 10 feature pairs, the average activation similarity is 0.91.

## E Feature family details

## E.1 Feature splitting structures

Figure 16 shows an example of a recurrent feature across SAE sizes that does not exhibit feature splitting. While the feature has extremely high activation and cosine similarity across every model pair, each model only learns 1 feature in this direction. In Figures 17a and 17b we show two examples of feature splitting across SAE16 - SAE32 - SAE64 trained on astro-ph . 17a appears to show canonical feature splitting as originally described in Bricken et al., 2023, with an increasing number of features splitting the semantic space at each SAE size. There exists a top-level 'periodicity'/'periodicity detection' feature universal to all three SAEs, with relatively high similarity to all other features, as well as novel, more granular features appearing in smaller SAEs, i.e. 'Quasiperiodic oscillations in blazars', which only appears in SAE64 and is highly dissimilar from other split features.

## O-type stars and their properties

Figure 16: Recurrent features across SAEs trained on astro-ph ; heatmap colored by activation similarity D ; all feature vector cosine similarities are &gt; 0 98 . .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-00a21234-cd73-4572-a976-a419cba113d0.jpg)

In contrast, 17b demonstrates nearest-neighbour features across models that do not exhibit semantically meaningful feature splitting. While the top-level 'Luminous Blue Variables (LBVs)" feature occurs at every model size, SAE64 also exhibits two additional features, 'Lemaitre-Tolman-Bondi (LTB) Models" and 'Lyman Break Galaxies (LBGs)", that are highly dissimilar to each other, the LBVs feature, and every other feature in the smaller models. We claim these are novel features, occurring for the first time in SAE64, and that SAE16/SAE32 do not learn features for any related higher-level concepts; instead, this grouping could be a spurious token-level correlation (LBV/LTB/LBG as similar acronyms).

Feature triplets In Figure 18a, we search for features that occur in n dirs = 3072 models and have highly aligned features in larger ( n dirs = 6144 9216 , ) models; we use this as a rough proxy for the number of re-occurring features. We find that significantly more features re-occur between models for higher k , with over 1100 feature triplets at &gt; 0 95 . cosine similarity for k = 16 ; as k increases, the number of triplets drops sharply.

Self-consistency In 18b we show the set overlap between nearest-neighbour matches between SAE16 and SAE64 found directly, and nearest-neighbour matches between SAE16 and SAE64 found via nearest-neighbour matches to SAE32. If features exhibit perfectly clean splitting geometry, then these two sets of SAE64 features should be consistent. However, we find that the distribution of set overlap is roughly bimodal; other than triplet features with perfect overlap, overlap generally ranges from 0 to 0.6. The vast majority of intersection = 1 sets are ≤ 3 features in size. This corroborates findings in 9 which suggests features across models with different k are not well-aligned.

## E.2 Feature family structure

We compute feature family sizes (including the parent), co-occurrence ratios ( R p, ( C ) , see section 4), and activation similarity ratios (computed identically to R p, ( C ) , just using activation similarities). Statistics for variants of cs.LG and astro-ph are shown in 19. We find a positive correlation (Spearman = 0.22) between R p, ( C ) and feature family interpretability.

We reproduce the projection method of Engels et al., 2024, running all documents through the SAE and ablating features not in the feature family, to produce Figure 20. Visualizing the resulting principal components confirms that the feature families we find do not represent manifolds or irreducible multi-dimensional structures. We can instead think of feature families as linear subspaces in the high-dimensional latent space; in fact, the component vectors can be seen in the lines of points representing documents only activating on one feature in the family.

In 4 we use n = 3 iterations of feature family construction. We select this hyper-parameter based off Figure 21. In the first 2-3 iterations, removing parent nodes and re-constructing features preferentially

appears in all SAEs

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-0a3b3b63-44be-4e6b-a012-278b43f2b36a.jpg)

(a) We find both recurrent features and novel features at every level (i.e. the top-level 'periodicity detection"/'periodicity" feature); heatmap colored by pairwise cosine similarity.

(b) While 'Luminous Blue Variables" is a recurrent feature in each SAE, SAE64 also exhibits 2 other nearest-neighbour features to 'Luminous Blue Variables" that are not semantically related; heatmap colored by pairwise cosine similarity.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-85601fa1-f38f-4f28-806e-f668933a0180.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-90170e7e-586f-4dcf-a2d5-5fe2d2cfb8d4.jpg)

(a) Number of features from the smallest SAE that re-occur in all SAEs, by cosine similarity threshold.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-3b292545-c218-4873-8b77-fa2d2bc9e17d.jpg)

(b) Overlap in the recovered SAE64 features, propagating nearest neighbors from SAE16SAE64 vs. SAE16-SAE32-SAE64.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-a4ed3db8-c31e-4714-afe3-702a22242e2c.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-f8449372-9644-4223-afaf-8e2dd8a8a08f.jpg)

(a)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-3388b918-2251-419f-958e-56a906d4d15f.jpg)

(b)

(c)

Figure 19: Feature families statistics (left: size; middle: activation similarity ratio; right: cooccurrence ratio, R p, ( C ) ); k = 64 , n dir = 9216 .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-006b4aa9-0744-40ff-a057-b236e7526220.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-e9436d22-8ccd-44d0-8c0c-f45d9831e264.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-621d43c9-0e95-4aa9-87c5-f82cfdbc01be.jpg)

Figure 20: PCA projections of 3 example feature families from SAE64; points are latent representations of activating examples, colored by average activation for in-family features in the top k .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-b6e50286-1d4c-4ab3-bef0-c041635124c2.jpg)

Figure 21: New feature families as a function of iteration; no deduplication is performed.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-e5eba7f6-db9f-4eab-b7f5-a6f061cb0100.jpg)

creates additional smaller families, suggesting iterations are necessary to fully explore the graph. But given the sparse co-occurrences ( C i,j &gt; . 0 1 ) used to build the graph, the number of additional feature families found at each iteration drops off steeply after n = 3 .

## E.3 Feature family interpretability

We show example feature families and their interpretability scores in Figure 22.

## F Exploring learned decoder weight matrices

Encoder and decoder representations Figure 23 reveals an intriguing relationship between feature distinctiveness and the similarity of encoder and decoder representations in our sparse autoencoder. In an ideal scenario with orthogonal features, encoder and decoder vectors would be identical, as the optimal detection direction (encoder) would align perfectly with the representation direction (decoder). This is because orthogonal features can be uniquely identified without interference. However, in our high-dimensional space with more features than dimensions, perfect orthogonality is impossible due to superposition.

The right panel of Figure 23 shows a negative correlation between a feature's decoder-encoder cosine similarity and its maximum similarity with other features. Features more orthogonal to others (lower maximum similarity) tend to have more similar encoder and decoder representations. This aligns with intuition: for more isolated features, the encoder's detection direction can closely match the decoder's representation direction. Conversely, features with higher similarity to others require the encoder to adopt a more differentiated detection strategy to minimise interference, resulting in lower encoder-decoder similarity. The left panel, showing a mean cosine similarity of 0.57 between corresponding encoder and decoder vectors, further emphasises this departure from orthogonality. This phenomenon points to the importance of untied weights in sparse autoencoders.

Clustering feature vectors Motivated by structure in the feature activation graph, we explore whether similar structure can be found in the decoder weight matrix W itself. Gao et al., 2024 find 2 such clusters; we reproduce their method across our embeddings and SAEs, permuting the left singular vectors U of W using a one-dimensional UMAP. We also experiment with permuting U and W using reverse Cuthill-McKee. We do not find any meaningful block diagonal structure or clustering in W .

## G Iterative encoding optimisation

We noted in Section 5 that intervening on a feature by up- or down-weighting its hidden representation and then decoding is equivalent to directly adding the scaled feature vector to the final embedding.

Individual Features

Family F1 (base)

Figure 22: High-quality (top) and low-quality (bottom) feature families, scored through automated interpretability; radar charts show Pearson correlation scores for individual features (vertices) and the overall family (dashed line). While high-quality feature families truly have shared meaning, low-quality families appear to be mostly spurious and are not interpretable through short descriptions.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-cc20529e-d969-4864-b7b5-7fd29cef0907.jpg)

Figure 23: (Left) Cosine similarities between the encoder row and corresponding decoder column for SAE64 ( cs.LG ). The mean cosine similarity is 0.57, suggesting that encoder and decoder features are rather different, agreeing with Nanda (2023). (Right) We notice a slight negative correlation between a feature's decoder-encoder cosine similarity, and its maximum similarity with other features, possibly suggesting that features that are furthest removed from all other features in embedding space can have more similar corresponding decoders and encoder projections.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-d6a05fb4-672c-4e62-b997-f2608d82b4ad.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-f3839778-568b-486e-88c0-29546f464679.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-1332a6de-f383-4983-8963-11c332fa98a8.jpg)

Figure 24: UMAP density plots along with LLM generated labels for SAE16 (left) and SAE64 (right) for the astro-ph features.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-34d03c29-9959-454a-b686-98e41357d84a.jpg)

Figure 25: Distribution of maximum cosine similarity between a given feature vector and all other feature vectors, within the same SAE.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-d06b4198-1ab9-4bba-abd3-8581c25c7b72.jpg)

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-763c4801-41c0-43f9-904e-2d645c6f9df7.jpg)

Figure 26: Cosine similarity between the original query embedding and the modified query embedding, with different values of upweighting random zero features and downweighting random active features.

To demonstrate this equivalence, let's consider an intervention on feature i by an amount δ . The modified hidden representation is h ′ = h + δ e i , where e i is the i -th standard basis vector. Decoding this modified representation gives ˆ x ′ = W d h ′ = W d h + δW d e i = ˆ + x δ w i , where w i is the i -th column of W d . Thus, intervening on the hidden representation and then decoding is equivalent to directly adding the scaled feature vector to the original reconstruction.

We show in Figure 26 how cosine similarity between the original query embedding and the modified query embedding changes as we change the upweighting and downweighting strength for different features. Cosine similarity drops rapidly as soon as upweight or downweight exceeds 0.1.

There is an implicit challenge in SAE-based embedding interventions: the trade-off between steering strength and precision. When directly manipulating feature activations, we observed that strong interventions often led to unintended semantic shifts, activating correlated features and potentially moving the embedding far from the SAE's learned manifold. Our goal is to achieve precise semantic edits that express the desired feature strongly while minimising interference with unrelated features. To this end, we developed an iterative optimisation approach that leverages the SAE's learned feature space to find an optimal balance between these competing objectives.

Let x ∈ R d be the original embedding, f θ ( ) · the SAE encoder, and g ϕ ( ) · the SAE decoder. We define a target feature vector t ∈ R k representing the desired feature activations after intervention, where k is the number of active features in our SAE. The iterative latent optimisation aims to find optimised latents h ∗ that satisfy:

$$h ^ { * } = \arg \min _ { h ^ { \prime } } \left \{ \| f _ { \theta } ( g _ { \phi } ( { \mathbf h } ^ { \prime } ) ) - { \mathbf t } \| _ { 2 } ^ { 2 } \right \}$$

We solve this optimisation problem using gradient descent, starting from the initial latents h = f θ ( x ) and iteratively updating h ′ . We use the AdamW optimiser with a cosine annealing learning rate schedule.

̸

To evaluate the effectiveness of this approach, we compare it to a direct intervention method where we simply set the target feature to a specific value in the latent space. For each abstract in our dataset, we embed the abstract using an OpenAI embedding model to obtain x . We then encode the embedding to get initial latents h = f θ ( x ) . We randomly select a target feature i and target value v . We then apply both intervention methods: our iterative optimisation of h ′ as described above, with t i = v and t j = h j for j = i , and direct intervention: setting h ′ i = v and h ′ j = h j for j = i .

̸

Figure 27: Normalised MSE at each of 10 steps across the iterative latent optimisation process. Left: Setting a random zero feature to active. Right: Setting a random active feature to zero.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00657v2-figure-56852cd4-5614-43b6-962c-ebebeca050b5.jpg)

Figure 27 (left panel) shows the trajectory of normalised MSE during the iterative optimisation process, when setting a random zero feature to active. Similarly, the right panel shows the optimisation when setting a random active feature to zero. Normalised MSE improves in the former case but not the latter.
Charles O'Neill Christine Ye Kartheik Iyer John F. Wu
Disentangling Dense Embeddings with Sparse Autoencoders
Sparse autoencoders (SAEs) have shown promise in extracting interpretable
features from complex neural networks. We present one of the first applications
of SAEs to dense text embeddings from large language models, demonstrating
their effectiveness in disentangling semantic concepts. By training SAEs on
embeddings of over 420,000 scientific paper abstracts from computer science and
astronomy, we show that the resulting sparse representations maintain semantic
fidelity while offering interpretability. We analyse these learned features,
exploring their behaviour across different model capacities and introducing a
novel method for identifying ``feature families'' that represent related
concepts at varying levels of abstraction. To demonstrate the practical utility
of our approach, we show how these interpretable features can be used to
precisely steer semantic search, allowing for fine-grained control over query
semantics. This work bridges the gap between the semantic richness of dense
embeddings and the interpretability of sparse representations. We open source
our embeddings, trained sparse autoencoders, and interpreted features, as well
as a web app for exploring them.