content.raw: 🔍 Today's pick in Interpretability & Analysis of LMs: by 
content.type: text
content.value: 🔍 Today's pick in Interpretability & Analysis of LMs: by 
content.raw: @aadityasingh
content.type: mention
content.user: aadityasingh
content.raw:  T. Moskovitz, F. Hill, S. C. Y. Chan, A. M. Saxe (
content.type: text
content.value:  T. Moskovitz, F. Hill, S. C. Y. Chan, A. M. Saxe (
content.raw: @gatsbyunit
content.type: mention
content.user: gatsbyunit
content.raw: )
content.type: text
content.value: )
content.type: new_line
content.type: new_line
content.raw: This work proposes a new methodology inspired by optogenetics (dubbed "clamping") to perform targeted ablations during training to estimate the causal effect of specific interventions on mechanism formation.
content.type: text
content.value: This work proposes a new methodology inspired by optogenetics (dubbed "clamping") to perform targeted ablations during training to estimate the causal effect of specific interventions on mechanism formation.
content.type: new_line
content.type: new_line
content.raw: Authors use this approach to study the formation of induction heads training a 2L attention-only transformer to label examples via context information.
content.type: text
content.value: Authors use this approach to study the formation of induction heads training a 2L attention-only transformer to label examples via context information.
content.type: new_line
content.type: new_line
content.raw: Notable findings:
content.type: text
content.value: Notable findings:
content.type: new_line
content.type: new_line
content.raw: - The effects of induction heads are additive and redundant, with weaker heads compensating well for the ablation of a strong induction head in case the latter is ablated.
content.type: text
content.value: - The effects of induction heads are additive and redundant, with weaker heads compensating well for the ablation of a strong induction head in case the latter is ablated.
content.type: new_line
content.raw: - Competition between induction heads might emerge as a product of optimization pressure to converge faster, but it is not strictly necessary as all heads eventually learn to solve the task.
content.type: text
content.value: - Competition between induction heads might emerge as a product of optimization pressure to converge faster, but it is not strictly necessary as all heads eventually learn to solve the task.
content.type: new_line
content.raw: - Previous token heads (PTH) influence induction heads in a many-to-many fashion, with any PTH eliciting above-chance prediction from a subsequent induction head
content.type: text
content.value: - Previous token heads (PTH) influence induction heads in a many-to-many fashion, with any PTH eliciting above-chance prediction from a subsequent induction head
content.type: new_line
content.raw: - Three subcircuits for induction are identified, respectively mixing token-label information (1 + 2), matching the previous occurrence of the current class in the context (3qk + 4), and copying the label of the matched class (3v + 5).
content.type: text
content.value: - Three subcircuits for induction are identified, respectively mixing token-label information (1 + 2), matching the previous occurrence of the current class in the context (3qk + 4), and copying the label of the matched class (3v + 5).
content.type: new_line
content.raw: - The formation of induction heads is slowed down by a larger number of classes & labels, with more classes and more labels slowing down the formation of the matching and copying mechanisms, respectively. This may have implications when selecting a vocabulary size for LLMs: larger vocabularies lead to an increased compression ratio and longer contexts, but they might make copying more challenging by delaying the formation of induction heads.
content.type: text
content.value: - The formation of induction heads is slowed down by a larger number of classes & labels, with more classes and more labels slowing down the formation of the matching and copying mechanisms, respectively. This may have implications when selecting a vocabulary size for LLMs: larger vocabularies lead to an increased compression ratio and longer contexts, but they might make copying more challenging by delaying the formation of induction heads.
content.type: new_line
content.type: new_line
content.raw: 💻 Code: 
content.type: text
content.value: 💻 Code: 
content.href: https://github.com/aadityasingh/icl-dynamics
content.raw: https://github.com/aadityasingh/icl-dynamics
content.type: link
content.type: new_line
content.type: new_line
content.raw: 📄 Paper: 
content.type: text
content.value: 📄 Paper: 
content.label: What needs to go right for an induction head? A mechanistic study of
    in-context learning circuits and their formation (2404.07129)
content.raw: https://huggingface.co/papers/2404.07129
content.id: 2404.07129
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2404.07129
content.type: new_line
content.type: new_line
content.raw: 🔍 All daily picks: 
content.type: text
content.value: 🔍 All daily picks: 
content.href: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-ofc-lms-65ae3339949c5675d25de2f9
content.raw: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-ofc-lms-65ae3339949c5675d25de2f9
content.type: link
rawContent: 🔍 Today's pick in Interpretability & Analysis of LMs: by @aadityasingh T. Moskovitz, F. Hill, S. C. Y. Chan, A. M. Saxe (@gatsbyunit)
  
  This work proposes a new methodology inspired by optogenetics (dubbed "clamping") to perform targeted ablations during training to estimate the causal effect of specific interventions on mechanism formation.
  
  Authors use this approach to study the formation of induction heads training a 2L attention-only transformer to label examples via context information.
  
  Notable findings:
  
  - The effects of induction heads are additive and redundant, with weaker heads compensating well for the ablation of a strong induction head in case the latter is ablated.
  - Competition between induction heads might emerge as a product of optimization pressure to converge faster, but it is not strictly necessary as all heads eventually learn to solve the task.
  - Previous token heads (PTH) influence induction heads in a many-to-many fashion, with any PTH eliciting above-chance prediction from a subsequent induction head
  - Three subcircuits for induction are identified, respectively mixing token-label information (1 + 2), matching the previous occurrence of the current class in the context (3qk + 4), and copying the label of the matched class (3v + 5).
  - The formation of induction heads is slowed down by a larger number of classes & labels, with more classes and more labels slowing down the formation of the matching and copying mechanisms, respectively. This may have implications when selecting a vocabulary size for LLMs: larger vocabularies lead to an increased compression ratio and longer contexts, but they might make copying more challenging by delaying the formation of induction heads.
  
  💻 Code: https://github.com/aadityasingh/icl-dynamics
  
  📄 Paper: https://huggingface.co/papers/2404.07129
  
  🔍 All daily picks: https://huggingface.co/collections/gsarti/daily-picks-in-interpretability-and-analysis-ofc-lms-65ae3339949c5675d25de2f9
reactions.reaction: ❤️
reactions.users: javifer samusenps