title: Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting
    Rare Concepts in Foundation Models
summary: Understanding and mitigating the potential risks associated with foundation
  models (FMs) hinges on developing effective interpretability methods. Sparse
  Autoencoders (SAEs) have emerged as a promising tool for disentangling FM
  representations, but they struggle to capture rare, yet crucial concepts in the
  data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to
  illuminate these elusive dark matter features by focusing on specific
  subdomains. We present a practical recipe for training SSAEs, demonstrating the
  efficacy of dense retrieval for data selection and the benefits of Tilted
  Empirical Risk Minimization as a training objective to improve concept recall.
  Our evaluation of SSAEs on standard metrics, such as downstream perplexity and
  L_0 sparsity, show that they effectively capture subdomain tail concepts,
  exceeding the capabilities of general-purpose SAEs. We showcase the practical
  utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs
  achieve a 12.5\% increase in worst-group classification accuracy when applied
  to remove spurious gender information. SSAEs provide a powerful new lens for
  peering into the inner workings of FMs in subdomains.
paper_authors._id: 67298c6932c59bb502696f25
paper_authors.name: Aashiq Muhamed
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T16:31:59.109Z
paper_authors._id: 64755a83e0b188d3cb2579d8
paper_authors.avatarUrl: /avatars/2c50590905f4bd398a4c9991e1b4b5bb.svg
paper_authors.fullname: Aashiq Muhamed
paper_authors.type: user
paper_authors.user: aashiqmuhamed
paper_authors._id: 67298c6932c59bb502696f26
paper_authors.name: Mona Diab
paper_authors._id: 67298c6932c59bb502696f27
paper_authors.name: Virginia Smith
paper_title: Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting
    Rare Concepts in Foundation Models
paper_summary: Understanding and mitigating the potential risks associated with foundation
  models (FMs) hinges on developing effective interpretability methods. Sparse
  Autoencoders (SAEs) have emerged as a promising tool for disentangling FM
  representations, but they struggle to capture rare, yet crucial concepts in the
  data. We introduce Specialized Sparse Autoencoders (SSAEs), designed to
  illuminate these elusive dark matter features by focusing on specific
  subdomains. We present a practical recipe for training SSAEs, demonstrating the
  efficacy of dense retrieval for data selection and the benefits of Tilted
  Empirical Risk Minimization as a training objective to improve concept recall.
  Our evaluation of SSAEs on standard metrics, such as downstream perplexity and
  L_0 sparsity, show that they effectively capture subdomain tail concepts,
  exceeding the capabilities of general-purpose SAEs. We showcase the practical
  utility of SSAEs in a case study on the Bias in Bios dataset, where SSAEs
  achieve a 12.5\% increase in worst-group classification accuracy when applied
  to remove spurious gender information. SSAEs provide a powerful new lens for
  peering into the inner workings of FMs in subdomains.
paper_ai_keywords: Sparse Autoencoders (SAEs) Specialized Sparse Autoencoders (SSAEs) disentangling dense retrieval Tilted Empirical Risk Minimization downstream perplexity $L_0$ sparsity subdomain tail concepts worst-group classification accuracy spurious gender information