title: The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax
    Mimicry
summary: Linear attentions have shown potential for improving Transformer efficiency,
  reducing attention's quadratic complexity to linear in sequence length. This
  holds exciting promise for (1) training linear Transformers from scratch, (2)
  "finetuned-conversion" of task-specific Transformers into linear versions that
  recover task performance, and (3) "pretrained-conversion" of Transformers such
  as large language models into linear versions finetunable on downstream tasks.
  However, linear attentions often underperform standard softmax attention in
  quality. To close this performance gap, we find prior linear attentions lack
  key properties of softmax attention tied to good performance: low-entropy (or
  "spiky") weights and dot-product monotonicity. We further observe surprisingly
  simple feature maps that retain these properties and match softmax performance,
  but are inefficient to compute in linear attention. We thus propose Hedgehog, a
  learnable linear attention that retains the spiky and monotonic properties of
  softmax attention while maintaining linear complexity. Hedgehog uses simple
  trainable MLPs to produce attention weights mimicking softmax attention.
  Experiments show Hedgehog recovers over 99% of standard Transformer quality in
  train-from-scratch and finetuned-conversion settings, outperforming prior
  linear attentions up to 6 perplexity points on WikiText-103 with causal GPTs,
  and up to 8.7 GLUE score points on finetuned bidirectional BERTs. Hedgehog also
  enables pretrained-conversion. Converting a pretrained GPT-2 into a linear
  attention variant achieves state-of-the-art 16.7 perplexity on WikiText-103 for
  125M subquadratic decoder models. We finally turn a pretrained Llama-2 7B into
  a viable linear attention Llama. With low-rank adaptation, Hedgehog-Llama2 7B
  achieves 28.1 higher ROUGE-1 points over the base standard attention model,
  where prior linear attentions lead to 16.5 point drops.
paper_authors._id: 65c467e7d853e2702e4d0a2c
paper_authors.name: Michael Zhang
paper_authors._id: 65c467e7d853e2702e4d0a2d
paper_authors.name: Kush Bhatia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T12:00:18.881Z
paper_authors._id: 64de95cf6466b91172197512
paper_authors.avatarUrl: /avatars/1b7934ad28103494a534ab0bd591173d.svg
paper_authors.fullname: Kush Bhatia
paper_authors.type: user
paper_authors.user: kushb
paper_authors._id: 65c467e7d853e2702e4d0a2e
paper_authors.name: Hermann Kumbong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T12:00:26.070Z
paper_authors._id: 6525fd79e0094b7bf21c01e6
paper_authors.avatarUrl: /avatars/04616a2a0363213ed4e577e92cf8f1b8.svg
paper_authors.fullname: Hermann Kumbong
paper_authors.type: user
paper_authors.user: kumboh
paper_authors._id: 65c467e7d853e2702e4d0a2f
paper_authors.name: Christopher Ré
paper_title: The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax
    Mimicry
paper_summary: Linear attentions have shown potential for improving Transformer efficiency,
  reducing attention's quadratic complexity to linear in sequence length. This
  holds exciting promise for (1) training linear Transformers from scratch, (2)
  "finetuned-conversion" of task-specific Transformers into linear versions that
  recover task performance, and (3) "pretrained-conversion" of Transformers such
  as large language models into linear versions finetunable on downstream tasks.
  However, linear attentions often underperform standard softmax attention in
  quality. To close this performance gap, we find prior linear attentions lack
  key properties of softmax attention tied to good performance: low-entropy (or
  "spiky") weights and dot-product monotonicity. We further observe surprisingly
  simple feature maps that retain these properties and match softmax performance,
  but are inefficient to compute in linear attention. We thus propose Hedgehog, a
  learnable linear attention that retains the spiky and monotonic properties of
  softmax attention while maintaining linear complexity. Hedgehog uses simple
  trainable MLPs to produce attention weights mimicking softmax attention.
  Experiments show Hedgehog recovers over 99% of standard Transformer quality in
  train-from-scratch and finetuned-conversion settings, outperforming prior
  linear attentions up to 6 perplexity points on WikiText-103 with causal GPTs,
  and up to 8.7 GLUE score points on finetuned bidirectional BERTs. Hedgehog also
  enables pretrained-conversion. Converting a pretrained GPT-2 into a linear
  attention variant achieves state-of-the-art 16.7 perplexity on WikiText-103 for
  125M subquadratic decoder models. We finally turn a pretrained Llama-2 7B into
  a viable linear attention Llama. With low-rank adaptation, Hedgehog-Llama2 7B
  achieves 28.1 higher ROUGE-1 points over the base standard attention model,
  where prior linear attentions lead to 16.5 point drops.
paper_ai_keywords: linear attentions Transformer sequence length softmax attention low-entropy spiky weights dot-product monotonicity feature maps Hedgehog MLPs attention weights perplexity points WikiText-103 causal GPTs GLUE score points bidirectional BERTs pretrained GPT-2 subquadratic decoder models pretrained Llama-2 low-rank adaptation Hedgehog-Llama2 ROUGE-1 points