Body: <p>I'm currently deeply invested in the <a href="https://transformer-circuits.pub" rel="nofollow noreferrer">Transformer Circuits thread</a> in parallel with <a href="https://www.youtube.com/watch?v=9-Jl0dxWQs8" rel="nofollow noreferrer">3blue1brown's videos</a> (chapter 7 on the MLP layer was released a day or two ago) to gain a better theoretical understanding of transformers, in particular relating to mechanistic interpretability. As I don't necessarily have a full CS/ML background, I sometimes don't really know how and where to ask for clarifications.</p>
  <p>One issue I'm wondering about whether the residual stream can be &quot;read&quot; (in the sense of unembedded) at various steps (before/after self-attention, before/after MLP, in-between layers) and whether that would produce meaningful or at least interesting tokens. My intuition or current understanding is that at first, unembedding the token's residual stream would produce the token itself, successive steps of attention and MLP further and further nudging the unembedding from a (nearly) 1-hot token encoding to the softmax next-token prediction.</p>
  <p>Is this intuition at least partially correct?</p>
  <p>More broadly, since these authors often consider toy models, is there something like GPT-2-mini with published weights available online where I could manually &quot;investigate&quot; the embeddings in the residual stream?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-04T19:50:37.490Z
Comments.Text: Had I only read [Neel Nanda's](https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-extremely-opinionated-annotated-list-of-my-favourite#Bonus) post to the end, I would have found the [LogitLens](https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens) which seems to be exactly what I'm looking for, but I'm happy to receive other suggestions.
Comments.UserDisplayName: Jonas Zaugg
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-09-05T08:26:24.113Z
Comments.Text: FYI, there's also [OpenAI's Transformer Debugger](https://github.com/openai/transformer-debugger) and [this one](https://www.lesswrong.com/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream), both based on GPT-2
Comments.UserDisplayName: Jonas Zaugg
Tags: <transformer><gpt><explainable-ai>
Title: Are there freely available pre-trained "toy models" of transformers suitable for inspecting residual stream?
UserDisplayName: Jonas Zaugg
ThreadText: # Are there freely available pre-trained "toy models" of transformers suitable for inspecting residual stream?
  
  (Asked by: username_0 on 2024-09-04)
  
  I'm currently deeply invested in the [Transformer Circuits thread](https://transformer-circuits.pub) in parallel with [3blue1brown's videos](https://www.youtube.com/watch?v=9-Jl0dxWQs8) (chapter 7 on the MLP layer was released a day or two ago) to gain a better theoretical understanding of transformers, in particular relating to mechanistic interpretability. As I don't necessarily have a full CS/ML background, I sometimes don't really know how and where to ask for clarifications.
  
  One issue I'm wondering about whether the residual stream can be "read" (in the sense of unembedded) at various steps (before/after self-attention, before/after MLP, in-between layers) and whether that would produce meaningful or at least interesting tokens. My intuition or current understanding is that at first, unembedding the token's residual stream would produce the token itself, successive steps of attention and MLP further and further nudging the unembedding from a (nearly) 1-hot token encoding to the softmax next-token prediction.
  
  Is this intuition at least partially correct?
  
  More broadly, since these authors often consider toy models, is there something like GPT-2-mini with published weights available online where I could manually "investigate" the embeddings in the residual stream?
  
  **Tags:** transformer, gpt, explainable-ai
  > **Comments:**
  > **username_0**: Had I only read [Neel Nanda's](https://www.alignmentforum.org/posts/NfFST5Mio7BCAQHPA/an-extremely-opinionated-annotated-list-of-my-favourite#Bonus) post to the end, I would have found the [LogitLens](https://www.alignmentforum.org/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens) which seems to be exactly what I'm looking for, but I'm happy to receive other suggestions.  
  > **username_0**: FYI, there's also [OpenAI's Transformer Debugger](https://github.com/openai/transformer-debugger) and [this one](https://www.lesswrong.com/posts/f9EgfLSurAiqRJySD/open-source-sparse-autoencoders-for-all-residual-stream), both based on GPT-2  
  
  ---
  
  ## 0 answers
  
  