title: MoA: Mixture of Sparse Attention for Automatic Large Language Model
    Compression
summary: Sparse attention can effectively mitigate the significant memory and
  throughput demands of Large Language Models (LLMs) in long contexts. Existing
  methods typically employ a uniform sparse attention mask, applying the same
  sparse pattern across different attention heads and input lengths. However,
  this uniform approach fails to capture the diverse attention patterns inherent
  in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this
  challenge, we propose the Mixture of Attention (MoA), which automatically
  tailors distinct sparse attention configurations to different heads and layers.
  MoA constructs and navigates a search space of various attention patterns and
  their scaling rules relative to input sequence lengths. It profiles the model,
  evaluates potential configurations, and pinpoints the optimal sparse attention
  compression plan. MoA adapts to varying input sizes, revealing that some
  attention heads expand their focus to accommodate longer sequences, while other
  heads consistently concentrate on fixed-length local contexts. Experiments show
  that MoA increases the effective context length by 3.9times with the same
  average attention span, boosting retrieval accuracy by 1.5-7.1times over the
  uniform-attention baseline across Vicuna-7B, Vicuna-13B, and Llama3-8B models.
  Moreover, MoA narrows the capability gaps between sparse and dense models,
  reducing the maximum relative performance drop from 9%-36% to within 5%
  across two long-context understanding benchmarks. MoA achieves a
  1.2-1.4times GPU memory reduction and boosts decode throughput by 5.5-6.7
  times for 7B and 13B dense models on a single GPU, with minimal impact on
  performance.
paper_authors._id: 667e3ffc1127552103ab356e
paper_authors.name: Tianyu Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T09:12:56.556Z
paper_authors._id: 6445fd9ba56444c355dcbcba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6445fd9ba56444c355dcbcba/NCyRCD-MK-yA0_qY6I2y0.png
paper_authors.fullname: Tianyu Fu
paper_authors.type: user
paper_authors.user: fuvty
paper_authors._id: 667e3ffc1127552103ab356f
paper_authors.name: Haofeng Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-15T09:01:11.739Z
paper_authors._id: 644950b316425b79ffee5f13
paper_authors.avatarUrl: /avatars/b87d20dd659b727ec7e2d5ee4fda1852.svg
paper_authors.fullname: Jason Huang
paper_authors.type: user
paper_authors.user: jason-huang03
paper_authors._id: 667e3ffc1127552103ab3570
paper_authors.name: Xuefei Ning
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-28T08:02:43.467Z
paper_authors._id: 641031b1a78453b8d96b8420
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678782881444-noauth.jpeg
paper_authors.fullname: Xuefei Ning
paper_authors.type: user
paper_authors.user: Foxfi
paper_authors._id: 667e3ffc1127552103ab3571
paper_authors.name: Genghan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T12:32:53.942Z
paper_authors._id: 65a76ff1e504d9738d636217
paper_authors.avatarUrl: /avatars/26bf5e3f19057835ee95d72c24904d77.svg
paper_authors.fullname: Genghan Zhang
paper_authors.type: user
paper_authors.user: Genghan
paper_authors._id: 667e3ffc1127552103ab3572
paper_authors.name: Boju Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T12:33:02.302Z
paper_authors._id: 6503e40893574a897182f27f
paper_authors.avatarUrl: /avatars/2aff6e564565bd243fdb5dfb2f1ef545.svg
paper_authors.fullname: Boju Chen
paper_authors.type: user
paper_authors.user: Herdsman
paper_authors._id: 667e3ffc1127552103ab3573
paper_authors.name: Tianqi Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T12:33:08.192Z
paper_authors._id: 662c3a46b7c202c0842ab92e
paper_authors.avatarUrl: /avatars/d57cb6de6da34ef2b54ed434ae752d4d.svg
paper_authors.fullname: Tianqi Wu
paper_authors.type: user
paper_authors.user: wutianqidx
paper_authors._id: 667e3ffc1127552103ab3574
paper_authors.name: Hongyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T12:33:49.980Z
paper_authors._id: 64e5414fe56d371e42da69c4
paper_authors.avatarUrl: /avatars/7d4a101b0338410bc736e04d7a611aca.svg
paper_authors.fullname: Hongyi Wang
paper_authors.type: user
paper_authors.user: hwang595
paper_authors._id: 667e3ffc1127552103ab3575
paper_authors.name: Zixiao Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T12:34:01.780Z
paper_authors._id: 652392b46b01183a67042fc3
paper_authors.avatarUrl: /avatars/646e455e6bd39bbbf7abc8e3f3c8f225.svg
paper_authors.fullname: Huang Zixiao
paper_authors.type: user
paper_authors.user: Hzxx
paper_authors._id: 667e3ffc1127552103ab3576
paper_authors.name: Shiyao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T12:44:32.414Z
paper_authors._id: 64cb1a9fb704b851478bcd5b
paper_authors.avatarUrl: /avatars/b35ec1320adc058bff871f14971485ba.svg
paper_authors.fullname: Shiyao Li
paper_authors.type: user
paper_authors.user: LSY-noya
paper_authors._id: 667e3ffc1127552103ab3577
paper_authors.name: Shengen Yan
paper_authors._id: 667e3ffc1127552103ab3578
paper_authors.name: Guohao Dai
paper_authors._id: 667e3ffc1127552103ab3579
paper_authors.name: Huazhong Yang
paper_authors._id: 667e3ffc1127552103ab357a
paper_authors.name: Yu Wang
paper_title: MoA: Mixture of Sparse Attention for Automatic Large Language Model
    Compression
paper_summary: Sparse attention can effectively mitigate the significant memory and
  throughput demands of Large Language Models (LLMs) in long contexts. Existing
  methods typically employ a uniform sparse attention mask, applying the same
  sparse pattern across different attention heads and input lengths. However,
  this uniform approach fails to capture the diverse attention patterns inherent
  in LLMs, ignoring their distinct accuracy-latency trade-offs. To address this
  challenge, we propose the Mixture of Attention (MoA), which automatically
  tailors distinct sparse attention configurations to different heads and layers.
  MoA constructs and navigates a search space of various attention patterns and
  their scaling rules relative to input sequence lengths. It profiles the model,
  evaluates potential configurations, and pinpoints the optimal sparse attention
  compression plan. MoA adapts to varying input sizes, revealing that some
  attention heads expand their focus to accommodate longer sequences, while other
  heads consistently concentrate on fixed-length local contexts. Experiments show
  that MoA increases the effective context length by 3.9times with the same
  average attention span, boosting retrieval accuracy by 1.5-7.1times over the
  uniform-attention baseline across Vicuna-7B, Vicuna-13B, and Llama3-8B models.
  Moreover, MoA narrows the capability gaps between sparse and dense models,
  reducing the maximum relative performance drop from 9%-36% to within 5%
  across two long-context understanding benchmarks. MoA achieves a
  1.2-1.4times GPU memory reduction and boosts decode throughput by 5.5-6.7
  times for 7B and 13B dense models on a single GPU, with minimal impact on
  performance.
paper_ai_keywords: Sparse attention Large Language Models (LLMs) Attention heads Input lengths Mixture of Attention (MoA) Attention patterns Scaling rules Input sequence lengths Optimal sparse attention Context length Attention span Retrieval accuracy Vicuna-7B Vicuna-13B Llama3-8B GPU memory reduction Decode throughput