title: The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
summary: Sparse attention offers a promising strategy to extend long-context
  capabilities in Transformer LLMs, yet its viability, its efficiency-accuracy
  trade-offs, and systematic scaling studies remain unexplored. To address this
  gap, we perform a careful comparison of training-free sparse attention methods
  at varying model scales, sequence lengths, and sparsity levels on a diverse
  collection of long-sequence tasks-including novel ones that rely on natural
  language while remaining controllable and easy to evaluate. Based on our
  experiments, we report a series of key findings: 1) an isoFLOPS analysis
  reveals that for very long sequences, larger and highly sparse models are
  preferable to smaller and dense ones. 2) The level of sparsity attainable while
  statistically guaranteeing accuracy preservation is higher during decoding than
  prefilling, and correlates with model size in the former. 3) There is no clear
  strategy that performs best across tasks and phases, with different units of
  sparsification or budget adaptivity needed for different scenarios. Even
  moderate sparsity levels often result in significant performance degradation on
  at least one task, highlighting that sparse attention is not a universal
  solution. 4) We introduce and validate novel scaling laws specifically tailored
  for sparse attention, providing evidence that our findings are likely to hold
  true beyond our range of experiments. Through these insights, we demonstrate
  that sparse attention is a key tool to enhance the capabilities of Transformer
  LLMs for processing longer sequences, but requires careful evaluation of
  trade-offs for performance-sensitive applications.
paper_authors._id: 680f2668db85fd31cd5080ff
paper_authors.name: Piotr Nawrot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:33:38.822Z
paper_authors._id: 640deb5d3c82bd463ee44735
paper_authors.avatarUrl: /avatars/0e748d7c91d97526b280e40ccb25c9e0.svg
paper_authors.fullname: Piotr Nawrot
paper_authors.type: user
paper_authors.user: pnawrot
paper_authors._id: 680f2668db85fd31cd508100
paper_authors.name: Robert Li
paper_authors._id: 680f2668db85fd31cd508101
paper_authors.name: Renjie Huang
paper_authors._id: 680f2668db85fd31cd508102
paper_authors.name: Sebastian Ruder
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:34:22.719Z
paper_authors._id: 624575789dc26c59bae6a19e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1648719221694-noauth.jpeg
paper_authors.fullname: Sebastian Ruder
paper_authors.type: user
paper_authors.user: ruder
paper_authors._id: 680f2668db85fd31cd508103
paper_authors.name: Kelly Marchisio
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-29T12:34:47.942Z
paper_authors._id: 66356b1e2fcf64c1548a1233
paper_authors.avatarUrl: /avatars/9a1fab7da6cd16e8bf2d104c399ae34f.svg
paper_authors.fullname: Kelly Marchisio
paper_authors.type: user
paper_authors.user: kellycohere
paper_authors._id: 680f2668db85fd31cd508104
paper_authors.name: Edoardo M. Ponti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-28T12:50:11.195Z
paper_authors._id: 60809ad44ad99100d63ce36a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619040921084-noauth.jpeg
paper_authors.fullname: Edoardo Maria Ponti
paper_authors.type: user
paper_authors.user: ducdauge
paper_title: The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
paper_summary: Sparse attention offers a promising strategy to extend long-context
  capabilities in Transformer LLMs, yet its viability, its efficiency-accuracy
  trade-offs, and systematic scaling studies remain unexplored. To address this
  gap, we perform a careful comparison of training-free sparse attention methods
  at varying model scales, sequence lengths, and sparsity levels on a diverse
  collection of long-sequence tasks-including novel ones that rely on natural
  language while remaining controllable and easy to evaluate. Based on our
  experiments, we report a series of key findings: 1) an isoFLOPS analysis
  reveals that for very long sequences, larger and highly sparse models are
  preferable to smaller and dense ones. 2) The level of sparsity attainable while
  statistically guaranteeing accuracy preservation is higher during decoding than
  prefilling, and correlates with model size in the former. 3) There is no clear
  strategy that performs best across tasks and phases, with different units of
  sparsification or budget adaptivity needed for different scenarios. Even
  moderate sparsity levels often result in significant performance degradation on
  at least one task, highlighting that sparse attention is not a universal
  solution. 4) We introduce and validate novel scaling laws specifically tailored
  for sparse attention, providing evidence that our findings are likely to hold
  true beyond our range of experiments. Through these insights, we demonstrate
  that sparse attention is a key tool to enhance the capabilities of Transformer
  LLMs for processing longer sequences, but requires careful evaluation of
  trade-offs for performance-sensitive applications.
paper_ai_summary: Sparse attention methods enable extending long-context capabilities in Transformer LLMs with varying efficiency-accuracy trade-offs across different tasks and scales.
paper_ai_keywords: sparse attention Transformer LLMs isoFLOPS prefilling decoding sparsity levels scaling laws