title: Softpick: No Attention Sink, No Massive Activations with Rectified
    Softmax
summary: We introduce softpick, a rectified, not sum-to-one, drop-in replacement for
  softmax in transformer attention mechanisms that eliminates attention sink and
  massive activations. Our experiments with 340M parameter models demonstrate
  that softpick maintains performance parity with softmax on standard benchmarks
  while achieving 0% sink rate. The softpick transformer produces hidden states
  with significantly lower kurtosis (340 vs 33,510) and creates sparse attention
  maps (46.97% sparsity). Models using softpick consistently outperform softmax
  when quantized, with particularly pronounced advantages at lower bit
  precisions. Our analysis and discussion shows how softpick has the potential to
  open new possibilities for quantization, low-precision training, sparsity
  optimization, pruning, and interpretability. Our code is available at
  https://github.com/zaydzuhri/softpick-attention.
paper_authors._id: 6812e473060494e99e4c5116
paper_authors.name: Zayd M. K. Zuhri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:35:00.473Z
paper_authors._id: 60cf8a354061635e43b28f60
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60cf8a354061635e43b28f60/o8gv9mG5cvnopJZJnNibi.jpeg
paper_authors.fullname: Zayd Muhammad Kawakibi Zuhri
paper_authors.type: user
paper_authors.user: zaydzuhri
paper_authors._id: 6812e473060494e99e4c5117
paper_authors.name: Erland Hilman Fuadi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-02T06:34:58.163Z
paper_authors._id: 6239ac869895e5c2a4345131
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6239ac869895e5c2a4345131/QErVhX2EdKUSg06eRrU7L.jpeg
paper_authors.fullname: Edd
paper_authors.type: user
paper_authors.user: Erland
paper_authors._id: 6812e473060494e99e4c5118
paper_authors.name: Alham Fikri Aji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-05T08:00:02.817Z
paper_authors._id: 61a4dc053205e107691e0d82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61a4dc053205e107691e0d82/BESoEHlHYXstXudh6dOdT.jpeg
paper_authors.fullname: Alham Fikri Aji
paper_authors.type: user
paper_authors.user: afaji
paper_title: Softpick: No Attention Sink, No Massive Activations with Rectified
    Softmax
paper_summary: We introduce softpick, a rectified, not sum-to-one, drop-in replacement for
  softmax in transformer attention mechanisms that eliminates attention sink and
  massive activations. Our experiments with 340M parameter models demonstrate
  that softpick maintains performance parity with softmax on standard benchmarks
  while achieving 0% sink rate. The softpick transformer produces hidden states
  with significantly lower kurtosis (340 vs 33,510) and creates sparse attention
  maps (46.97% sparsity). Models using softpick consistently outperform softmax
  when quantized, with particularly pronounced advantages at lower bit
  precisions. Our analysis and discussion shows how softpick has the potential to
  open new possibilities for quantization, low-precision training, sparsity
  optimization, pruning, and interpretability. Our code is available at
  https://github.com/zaydzuhri/softpick-attention.
paper_ai_summary: Softpick is a drop-in replacement for softmax in transformer attention mechanisms that improves performance, reduces activation sinks, and achieves sparsity, particularly beneficial for quantization and low-precision training.
paper_ai_keywords: softmax softpick attention mechanisms attention sink kurtosis sparse attention maps quantization low-precision training sparsity optimization pruning interpretability