title: Train Sparse Autoencoders Efficiently by Utilizing Features Correlation
summary: Sparse Autoencoders (SAEs) have demonstrated significant promise in
  interpreting the hidden states of language models by decomposing them into
  interpretable latent directions. However, training SAEs at scale remains
  challenging, especially when large dictionary sizes are used. While decoders
  can leverage sparse-aware kernels for efficiency, encoders still require
  computationally intensive linear operations with large output dimensions. To
  address this, we propose KronSAE, a novel architecture that factorizes the
  latent representation via Kronecker product decomposition, drastically reducing
  memory and computational overhead. Furthermore, we introduce mAND, a
  differentiable activation function approximating the binary AND operation,
  which improves interpretability and performance in our factorized framework.
paper_authors._id: 68385ed47d00cf0a04845b23
paper_authors.name: Vadim Kurochkin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:45.632Z
paper_authors._id: 63ef2bc2cdd847640245a657
paper_authors.avatarUrl: /avatars/cdac5fa9a4e69bb84794a90f9598be97.svg
paper_authors.fullname: Vadim Kurochkin
paper_authors.type: user
paper_authors.user: Vadim21221
paper_authors._id: 68385ed47d00cf0a04845b24
paper_authors.name: Yaroslav Aksenov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T13:40:04.628Z
paper_authors._id: 63ed5676684767daecac6f8a
paper_authors.avatarUrl: /avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
paper_authors.fullname: Yaroslav Aksenov
paper_authors.type: user
paper_authors.user: yaraksen
paper_authors._id: 68385ed47d00cf0a04845b25
paper_authors.name: Daniil Laptev
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:55.867Z
paper_authors._id: 634c5f8cfb80cc6bcaf42c03
paper_authors.avatarUrl: /avatars/1f37db0e70cbaf9707f4c8cbcee37ca0.svg
paper_authors.fullname: Daniil Laptev
paper_authors.type: user
paper_authors.user: dlaptev
paper_authors._id: 68385ed47d00cf0a04845b26
paper_authors.name: Daniil Gavrilov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:48.683Z
paper_authors._id: 62a9c8edc19f92ae443ab37f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
paper_authors.fullname: Daniil Gavrilov
paper_authors.type: user
paper_authors.user: kefirski
paper_authors._id: 68385ed47d00cf0a04845b27
paper_authors.name: Nikita Balagansky
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-02T07:47:52.915Z
paper_authors._id: 60b364e7f88532cd79eaff7b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
paper_authors.fullname: Nikita Balagansky
paper_authors.type: user
paper_authors.user: elephantmipt
paper_title: Train Sparse Autoencoders Efficiently by Utilizing Features Correlation
paper_summary: Sparse Autoencoders (SAEs) have demonstrated significant promise in
  interpreting the hidden states of language models by decomposing them into
  interpretable latent directions. However, training SAEs at scale remains
  challenging, especially when large dictionary sizes are used. While decoders
  can leverage sparse-aware kernels for efficiency, encoders still require
  computationally intensive linear operations with large output dimensions. To
  address this, we propose KronSAE, a novel architecture that factorizes the
  latent representation via Kronecker product decomposition, drastically reducing
  memory and computational overhead. Furthermore, we introduce mAND, a
  differentiable activation function approximating the binary AND operation,
  which improves interpretability and performance in our factorized framework.
paper_ai_summary: KronSAE, a novel architecture using Kronecker product decomposition, enhances efficiency in training Sparse Autoencoders, while mAND, a differentiable binary AND function, improves interpretability and performance.
paper_ai_keywords: Sparse Autoencoders Kronecker product decomposition latent representation mAND differentiable activation function