title: Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
summary: Obtaining human-interpretable explanations of large, general-purpose language
  models is an urgent goal for AI safety. However, it is just as important that
  our interpretability methods are faithful to the causal dynamics underlying
  model behavior and able to robustly generalize to unseen inputs. Distributed
  Alignment Search (DAS) is a powerful gradient descent method grounded in a
  theory of causal abstraction that uncovered perfect alignments between
  interpretable symbolic algorithms and small deep learning models fine-tuned for
  specific tasks. In the present paper, we scale DAS significantly by replacing
  the remaining brute-force search steps with learned parameters -- an approach
  we call DAS. This enables us to efficiently search for interpretable causal
  structure in large language models while they follow instructions. We apply DAS
  to the Alpaca model (7B parameters), which, off the shelf, solves a simple
  numerical reasoning problem. With DAS, we discover that Alpaca does this by
  implementing a causal model with two interpretable boolean variables.
  Furthermore, we find that the alignment of neural representations with these
  variables is robust to changes in inputs and instructions. These findings mark
  a first step toward deeply understanding the inner-workings of our largest and
  most widely deployed language models.
paper_authors._id: 6462f224d1ccd517f46a807f
paper_authors.name: Zhengxuan Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-17T08:04:16.333Z
paper_authors._id: 604fa06d4f9b833fc41c75cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650432433568-604fa06d4f9b833fc41c75cf.png
paper_authors.fullname: Zhengxuan Wu
paper_authors.type: user
paper_authors.user: zhengxuanzenwu
paper_authors._id: 6462f224d1ccd517f46a8080
paper_authors.name: Atticus Geiger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T15:31:48.092Z
paper_authors._id: 627b2d0527dc4650b62eef42
paper_authors.avatarUrl: /avatars/e70381850f5657b54e90f5539f3d74eb.svg
paper_authors.fullname: Atticus Geiger
paper_authors.type: user
paper_authors.user: atticusg
paper_authors._id: 6462f224d1ccd517f46a8081
paper_authors.name: Christopher Potts
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-16T16:22:39.048Z
paper_authors._id: 62606790e3dd8b49f360f050
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650485126923-noauth.png
paper_authors.fullname: Chris Potts
paper_authors.type: user
paper_authors.user: cgpotts
paper_authors._id: 6462f224d1ccd517f46a8082
paper_authors.name: Noah D. Goodman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:57:48.804Z
paper_authors._id: 67321274c1f20c742bcf7a8d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/ltcQhre6eDRVzn6Vbbyhu.png
paper_authors.fullname: Noah D. Goodman
paper_authors.type: user
paper_authors.user: ngoodman
paper_title: Interpretability at Scale: Identifying Causal Mechanisms in Alpaca
paper_summary: Obtaining human-interpretable explanations of large, general-purpose language
  models is an urgent goal for AI safety. However, it is just as important that
  our interpretability methods are faithful to the causal dynamics underlying
  model behavior and able to robustly generalize to unseen inputs. Distributed
  Alignment Search (DAS) is a powerful gradient descent method grounded in a
  theory of causal abstraction that uncovered perfect alignments between
  interpretable symbolic algorithms and small deep learning models fine-tuned for
  specific tasks. In the present paper, we scale DAS significantly by replacing
  the remaining brute-force search steps with learned parameters -- an approach
  we call DAS. This enables us to efficiently search for interpretable causal
  structure in large language models while they follow instructions. We apply DAS
  to the Alpaca model (7B parameters), which, off the shelf, solves a simple
  numerical reasoning problem. With DAS, we discover that Alpaca does this by
  implementing a causal model with two interpretable boolean variables.
  Furthermore, we find that the alignment of neural representations with these
  variables is robust to changes in inputs and instructions. These findings mark
  a first step toward deeply understanding the inner-workings of our largest and
  most widely deployed language models.
paper_ai_summary: DAS, an enhanced gradient descent method, identifies interpretable causal structures in large language models like Alpaca by aligning neural representations with symbolic boolean variables.
paper_ai_keywords: Distributed Alignment Search (DAS) gradient descent causal abstraction interpretable symbolic algorithms fine-tuned models causal structure large language models Alpaca model numerical reasoning problem boolean variables neural representations