Softpick: No Attention Sink, No Massive Activations with Rectified
  Softmax
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for
softmax in transformer attention mechanisms that eliminates attention sink and
massive activations. Our experiments with 340M parameter models demonstrate
that softpick maintains performance parity with softmax on standard benchmarks
while achieving 0% sink rate. The softpick transformer produces hidden states
with significantly lower kurtosis (340 vs 33,510) and creates sparse attention
maps (46.97% sparsity). Models using softpick consistently outperform softmax
when quantized, with particularly pronounced advantages at lower bit
precisions. Our analysis and discussion shows how softpick has the potential to
open new possibilities for quantization, low-precision training, sparsity
optimization, pruning, and interpretability. Our code is available at
https://github.com/zaydzuhri/softpick-attention.
6812e473060494e99e4c5116
Zayd M. K. Zuhri
claimed_verified
2025-05-02T06:35:00.473Z
60cf8a354061635e43b28f60
https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg
Zayd Muhammad Kawakibi Zuhri
user
zaydzuhri
6812e473060494e99e4c5117
Erland Hilman Fuadi
claimed_verified
2025-05-02T06:34:58.163Z
6239ac869895e5c2a4345131
https://cdn-avatars.huggingface.co/v1/production/uploads/6239ac869895e5c2a4345131/QErVhX2EdKUSg06eRrU7L.jpeg
Edd
user
Erland
6812e473060494e99e4c5118
Alham Fikri Aji
admin_assigned
2025-05-05T08:00:02.817Z
61a4dc053205e107691e0d82
https://cdn-avatars.huggingface.co/v1/production/uploads/61a4dc053205e107691e0d82/BESoEHlHYXstXudh6dOdT.jpeg
Alham Fikri Aji
user
afaji
Softpick: No Attention Sink, No Massive Activations with Rectified
  Softmax
We introduce softpick, a rectified, not sum-to-one, drop-in replacement for
softmax in transformer attention mechanisms that eliminates attention sink and
massive activations. Our experiments with 340M parameter models demonstrate
that softpick maintains performance parity with softmax on standard benchmarks
while achieving 0% sink rate. The softpick transformer produces hidden states
with significantly lower kurtosis (340 vs 33,510) and creates sparse attention
maps (46.97% sparsity). Models using softpick consistently outperform softmax
when quantized, with particularly pronounced advantages at lower bit
precisions. Our analysis and discussion shows how softpick has the potential to
open new possibilities for quantization, low-precision training, sparsity
optimization, pruning, and interpretability. Our code is available at
https://github.com/zaydzuhri/softpick-attention.
Softpick is a drop-in replacement for softmax in transformer attention mechanisms that improves performance, reduces activation sinks, and achieves sparsity, particularly beneficial for quantization and low-precision training.
softmax softpick attention mechanisms attention sink kurtosis sparse attention maps quantization low-precision training sparsity optimization pruning interpretability