Train Sparse Autoencoders Efficiently by Utilizing Features Correlation
Sparse Autoencoders (SAEs) have demonstrated significant promise in
interpreting the hidden states of language models by decomposing them into
interpretable latent directions. However, training SAEs at scale remains
challenging, especially when large dictionary sizes are used. While decoders
can leverage sparse-aware kernels for efficiency, encoders still require
computationally intensive linear operations with large output dimensions. To
address this, we propose KronSAE, a novel architecture that factorizes the
latent representation via Kronecker product decomposition, drastically reducing
memory and computational overhead. Furthermore, we introduce mAND, a
differentiable activation function approximating the binary AND operation,
which improves interpretability and performance in our factorized framework.
68385ed47d00cf0a04845b23
Vadim Kurochkin
claimed_verified
2025-06-02T07:47:45.632Z
63ef2bc2cdd847640245a657
/avatars/cdac5fa9a4e69bb84794a90f9598be97.svg
Vadim Kurochkin
user
Vadim21221
68385ed47d00cf0a04845b24
Yaroslav Aksenov
claimed_verified
2025-06-02T13:40:04.628Z
63ed5676684767daecac6f8a
/avatars/d0e4a715f9c3fb6d74c183bab751ec35.svg
Yaroslav Aksenov
user
yaraksen
68385ed47d00cf0a04845b25
Daniil Laptev
claimed_verified
2025-06-02T07:47:55.867Z
634c5f8cfb80cc6bcaf42c03
/avatars/1f37db0e70cbaf9707f4c8cbcee37ca0.svg
Daniil Laptev
user
dlaptev
68385ed47d00cf0a04845b26
Daniil Gavrilov
claimed_verified
2025-06-02T07:47:48.683Z
62a9c8edc19f92ae443ab37f
https://cdn-avatars.huggingface.co/v1/production/uploads/1669110208492-62a9c8edc19f92ae443ab37f.png
Daniil Gavrilov
user
kefirski
68385ed47d00cf0a04845b27
Nikita Balagansky
claimed_verified
2025-06-02T07:47:52.915Z
60b364e7f88532cd79eaff7b
https://cdn-avatars.huggingface.co/v1/production/uploads/1654185363389-60b364e7f88532cd79eaff7b.jpeg
Nikita Balagansky
user
elephantmipt
Train Sparse Autoencoders Efficiently by Utilizing Features Correlation
Sparse Autoencoders (SAEs) have demonstrated significant promise in
interpreting the hidden states of language models by decomposing them into
interpretable latent directions. However, training SAEs at scale remains
challenging, especially when large dictionary sizes are used. While decoders
can leverage sparse-aware kernels for efficiency, encoders still require
computationally intensive linear operations with large output dimensions. To
address this, we propose KronSAE, a novel architecture that factorizes the
latent representation via Kronecker product decomposition, drastically reducing
memory and computational overhead. Furthermore, we introduce mAND, a
differentiable activation function approximating the binary AND operation,
which improves interpretability and performance in our factorized framework.
KronSAE, a novel architecture using Kronecker product decomposition, enhances efficiency in training Sparse Autoencoders, while mAND, a differentiable binary AND function, improves interpretability and performance.
Sparse Autoencoders Kronecker product decomposition latent representation mAND differentiable activation function