title: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse
    Attention
summary: Long-context modeling is crucial for next-generation language models, yet the
  high computational cost of standard attention mechanisms poses significant
  computational challenges. Sparse attention offers a promising direction for
  improving efficiency while maintaining model capabilities. We present NSA, a
  Natively trainable Sparse Attention mechanism that integrates algorithmic
  innovations with hardware-aligned optimizations to achieve efficient
  long-context modeling. NSA employs a dynamic hierarchical sparse strategy,
  combining coarse-grained token compression with fine-grained token selection to
  preserve both global context awareness and local precision. Our approach
  advances sparse attention design with two key innovations: (1) We achieve
  substantial speedups through arithmetic intensity-balanced algorithm design,
  with implementation optimizations for modern hardware. (2) We enable end-to-end
  training, reducing pretraining computation without sacrificing model
  performance. As shown in Figure 1, experiments show the model pretrained with
  NSA maintains or exceeds Full Attention models across general benchmarks,
  long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves
  substantial speedups over Full Attention on 64k-length sequences across
  decoding, forward propagation, and backward propagation, validating its
  efficiency throughout the model lifecycle.
paper_authors._id: 67b43211d3c5f50aa9c03a2d
paper_authors.name: Jingyang Yuan
paper_authors._id: 67b43211d3c5f50aa9c03a2e
paper_authors.name: Huazuo Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:43:19.672Z
paper_authors._id: 64e370be59aa5366642ac329
paper_authors.avatarUrl: /avatars/0fa1eb6ac6c1aeff3e65bc86a6617f64.svg
paper_authors.fullname: Huazuo Gao
paper_authors.type: user
paper_authors.user: gaohuazuo
paper_authors._id: 67b43211d3c5f50aa9c03a2f
paper_authors.name: Damai Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:43:30.267Z
paper_authors._id: 659389f8de82e1ef7b9a8b13
paper_authors.avatarUrl: /avatars/896ed9f4cdbd317493b303d070b7e12a.svg
paper_authors.fullname: Damai Dai
paper_authors.type: user
paper_authors.user: DeepSeekDDM
paper_authors._id: 67b43211d3c5f50aa9c03a30
paper_authors.name: Junyu Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:43:36.295Z
paper_authors._id: 66e6c6372c78909baf44cdf8
paper_authors.avatarUrl: /avatars/458ea1d545d7c022b0463e7fbbd91db1.svg
paper_authors.fullname: Junyu Luo
paper_authors.type: user
paper_authors.user: junyuluo
paper_authors._id: 67b43211d3c5f50aa9c03a31
paper_authors.name: Liang Zhao
paper_authors._id: 67b43211d3c5f50aa9c03a32
paper_authors.name: Zhengyan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:44:02.477Z
paper_authors._id: 65654ed2219af7f841640f27
paper_authors.avatarUrl: /avatars/e6904b3479fc5e65ea1f752919ca8290.svg
paper_authors.fullname: Zhengyan Zhang
paper_authors.type: user
paper_authors.user: ZhengyanZhang
paper_authors._id: 67b43211d3c5f50aa9c03a33
paper_authors.name: Zhenda Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:44:16.691Z
paper_authors._id: 6797ca96e9e2793006a15110
paper_authors.avatarUrl: /avatars/2d393d6e5fc2e1a867f7fdd44e055a2f.svg
paper_authors.fullname: zhenda xie
paper_authors.type: user
paper_authors.user: Zhendaxie
paper_authors._id: 67b43211d3c5f50aa9c03a34
paper_authors.name: Y. X. Wei
paper_authors._id: 67b43211d3c5f50aa9c03a35
paper_authors.name: Lean Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:44:26.979Z
paper_authors._id: 650c509472afb1e60e6151ae
paper_authors.avatarUrl: /avatars/c16ab5053a586819dc2b965303215ff7.svg
paper_authors.fullname: Lean Wang
paper_authors.type: user
paper_authors.user: AdaHousman
paper_authors._id: 67b43211d3c5f50aa9c03a36
paper_authors.name: Zhiping Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:44:34.873Z
paper_authors._id: 66ab566e30c55e83b02aa050
paper_authors.avatarUrl: /avatars/62692be88b9ad34ad3f474fb0359ae20.svg
paper_authors.fullname: Zhiping Xiao
paper_authors.type: user
paper_authors.user: Shockzipper
paper_authors._id: 67b43211d3c5f50aa9c03a37
paper_authors.name: Yuqing Wang
paper_authors._id: 67b43211d3c5f50aa9c03a38
paper_authors.name: Chong Ruan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-18T16:45:33.988Z
paper_authors._id: 6398203609f12714ed1935c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
paper_authors.fullname: Chong Ruan
paper_authors.type: user
paper_authors.user: Chester111
paper_authors._id: 67b43211d3c5f50aa9c03a39
paper_authors.name: Ming Zhang
paper_authors._id: 67b43211d3c5f50aa9c03a3a
paper_authors.name: Wenfeng Liang
paper_authors._id: 67b43211d3c5f50aa9c03a3b
paper_authors.name: Wangding Zeng
paper_title: Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse
    Attention
paper_summary: Long-context modeling is crucial for next-generation language models, yet the
  high computational cost of standard attention mechanisms poses significant
  computational challenges. Sparse attention offers a promising direction for
  improving efficiency while maintaining model capabilities. We present NSA, a
  Natively trainable Sparse Attention mechanism that integrates algorithmic
  innovations with hardware-aligned optimizations to achieve efficient
  long-context modeling. NSA employs a dynamic hierarchical sparse strategy,
  combining coarse-grained token compression with fine-grained token selection to
  preserve both global context awareness and local precision. Our approach
  advances sparse attention design with two key innovations: (1) We achieve
  substantial speedups through arithmetic intensity-balanced algorithm design,
  with implementation optimizations for modern hardware. (2) We enable end-to-end
  training, reducing pretraining computation without sacrificing model
  performance. As shown in Figure 1, experiments show the model pretrained with
  NSA maintains or exceeds Full Attention models across general benchmarks,
  long-context tasks, and instruction-based reasoning. Meanwhile, NSA achieves
  substantial speedups over Full Attention on 64k-length sequences across
  decoding, forward propagation, and backward propagation, validating its
  efficiency throughout the model lifecycle.
paper_ai_summary: NSA, a trainable sparse attention mechanism, enhances long-context modeling efficiency without sacrificing performance, achieving improvements in speed and accuracy over full attention models.
paper_ai_keywords: sparse attention long-context modeling token compression token selection arithmetic intensity-balanced full attention end-to-end training