title: The Hydra Effect: Emergent Self-repair in Language Model Computations
summary: We investigate the internal structure of language model computations using
  causal analysis and demonstrate two motifs: (1) a form of adaptive computation
  where ablations of one attention layer of a language model cause another layer
  to compensate (which we term the Hydra effect) and (2) a counterbalancing
  function of late MLP layers that act to downregulate the maximum-likelihood
  token. Our ablation studies demonstrate that language model layers are
  typically relatively loosely coupled (ablations to one layer only affect a
  small number of downstream layers). Surprisingly, these effects occur even in
  language models trained without any form of dropout. We analyse these effects
  in the context of factual recall and consider their implications for
  circuit-level attribution in language models.
paper_authors._id: 64c8648dcbcae9f106cd10d6
paper_authors.name: Thomas McGrath
paper_authors._id: 64c8648dcbcae9f106cd10d7
paper_authors.name: Matthew Rahtz
paper_authors._id: 64c8648dcbcae9f106cd10d8
paper_authors.name: Janos Kramar
paper_authors._id: 64c8648dcbcae9f106cd10d9
paper_authors.name: Vladimir Mikulik
paper_authors._id: 64c8648dcbcae9f106cd10da
paper_authors.name: Shane Legg
paper_title: The Hydra Effect: Emergent Self-repair in Language Model Computations
paper_summary: We investigate the internal structure of language model computations using
  causal analysis and demonstrate two motifs: (1) a form of adaptive computation
  where ablations of one attention layer of a language model cause another layer
  to compensate (which we term the Hydra effect) and (2) a counterbalancing
  function of late MLP layers that act to downregulate the maximum-likelihood
  token. Our ablation studies demonstrate that language model layers are
  typically relatively loosely coupled (ablations to one layer only affect a
  small number of downstream layers). Surprisingly, these effects occur even in
  language models trained without any form of dropout. We analyse these effects
  in the context of factual recall and consider their implications for
  circuit-level attribution in language models.
paper_ai_summary: Analysis reveals adaptive computation and downregulation mechanisms in language models, showing loose coupling among layers even without dropout.
paper_ai_keywords: causal analysis adaptive computation attention layer Hydra effect MLP layers maximum-likelihood token downregulation circuit-level attribution