title: Does Circuit Analysis Interpretability Scale? Evidence from Multiple
    Choice Capabilities in Chinchilla
summary: Circuit analysis is a promising technique for understanding the
  internal mechanisms of language models. However, existing analyses are done in
  small models far from the state of the art. To address this, we present a case
  study of circuit analysis in the 70B Chinchilla model, aiming to test the
  scalability of circuit analysis. In particular, we study multiple-choice
  question answering, and investigate Chinchilla's capability to identify the
  correct answer label given knowledge of the correct answer text.
  We find that the existing techniques of logit attribution, attention pattern
  visualization, and activation patching naturally scale to Chinchilla, allowing
  us to identify and categorize a small set of `output nodes' (attention heads
  and MLPs).
    We further study the `correct letter' category of attention heads aiming to
  understand the semantics of their features, with mixed results. For normal
  multiple-choice question answers, we significantly compress the query, key and
  value subspaces of the head without loss of performance when operating on the
  answer labels for multiple-choice questions, and we show that the query and key
  subspaces represent an `Nth item in an enumeration' feature to at least some
  extent. However, when we attempt to use this explanation to understand the
  heads' behaviour on a more general distribution including randomized answer
  labels, we find that it is only a partial explanation, suggesting there is more
  to learn about the operation of `correct letter' heads on multiple choice
  question answering.
paper_authors._id: 64b7476c9f5987572c9b2cfa
paper_authors.name: Tom Lieberum
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-20T07:17:40.948Z
paper_authors._id: 632cb97554e2c512c8f7bb31
paper_authors.avatarUrl: /avatars/634be7f8a59b6cf639e9e6c5e7ca1fbd.svg
paper_authors.fullname: TomL
paper_authors.type: user
paper_authors.user: Aric
paper_authors._id: 64b7476c9f5987572c9b2cfb
paper_authors.name: Matthew Rahtz
paper_authors._id: 64b7476c9f5987572c9b2cfc
paper_authors.name: János Kramár
paper_authors._id: 64b7476c9f5987572c9b2cfd
paper_authors.name: Geoffrey Irving
paper_authors._id: 64b7476c9f5987572c9b2cfe
paper_authors.name: Rohin Shah
paper_authors._id: 64b7476c9f5987572c9b2cff
paper_authors.name: Vladimir Mikulik
paper_title: Does Circuit Analysis Interpretability Scale? Evidence from Multiple
    Choice Capabilities in Chinchilla
paper_summary: Circuit analysis is a promising technique for understanding the
  internal mechanisms of language models. However, existing analyses are done in
  small models far from the state of the art. To address this, we present a case
  study of circuit analysis in the 70B Chinchilla model, aiming to test the
  scalability of circuit analysis. In particular, we study multiple-choice
  question answering, and investigate Chinchilla's capability to identify the
  correct answer label given knowledge of the correct answer text.
  We find that the existing techniques of logit attribution, attention pattern
  visualization, and activation patching naturally scale to Chinchilla, allowing
  us to identify and categorize a small set of `output nodes' (attention heads
  and MLPs).
    We further study the `correct letter' category of attention heads aiming to
  understand the semantics of their features, with mixed results. For normal
  multiple-choice question answers, we significantly compress the query, key and
  value subspaces of the head without loss of performance when operating on the
  answer labels for multiple-choice questions, and we show that the query and key
  subspaces represent an `Nth item in an enumeration' feature to at least some
  extent. However, when we attempt to use this explanation to understand the
  heads' behaviour on a more general distribution including randomized answer
  labels, we find that it is only a partial explanation, suggesting there is more
  to learn about the operation of `correct letter' heads on multiple choice
  question answering.
paper_ai_keywords: logit attribution attention pattern visualization activation patching output nodes attention heads MLPs query subspace key subspace value subspace Nth item in an enumeration