title: BurstAttention: An Efficient Distributed Attention Framework for
    Extremely Long Sequences
summary: Effective attention modules have played a crucial role in the success of
  Transformer-based large language models (LLMs), but the quadratic time and
  memory complexities of these attention modules also pose a challenge when
  processing long sequences. One potential solution for the long sequence problem
  is to utilize distributed clusters to parallelize the computation of attention
  modules across multiple devices (e.g., GPUs). However, adopting a distributed
  approach inevitably introduces extra memory overheads to store local attention
  results and incurs additional communication costs to aggregate local results
  into global ones. In this paper, we propose a distributed attention framework
  named ``BurstAttention'' to optimize memory access and communication operations
  at both the global cluster and local device levels. In our experiments, we
  compare BurstAttention with other competitive distributed attention solutions
  for long sequence processing. The experimental results under different length
  settings demonstrate that BurstAttention offers significant advantages for
  processing long sequences compared with these competitive baselines, reducing
  40% communication overheads and achieving 2 X speedup during training 32K
  sequence length on 8 X A100.
paper_authors._id: 65f3c3208e0f2653cdef4f74
paper_authors.name: Sun Ao
paper_authors._id: 65f3c3208e0f2653cdef4f75
paper_authors.name: Weilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:24:24.183Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 65f3c3208e0f2653cdef4f76
paper_authors.name: Xu Han
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-03-15T03:40:17.986Z
paper_authors._id: 65b8bd1dd4f282b64711d049
paper_authors.avatarUrl: /avatars/4e507f0ef4f3420bcff2677fd8ee14e5.svg
paper_authors.fullname: Xu Han
paper_authors.type: user
paper_authors.user: SillyXu
paper_authors._id: 65f3c3208e0f2653cdef4f77
paper_authors.name: Cheng Yang
paper_authors._id: 65f3c3208e0f2653cdef4f78
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:25:58.934Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 65f3c3208e0f2653cdef4f79
paper_authors.name: Chuan Shi
paper_authors._id: 65f3c3208e0f2653cdef4f7a
paper_authors.name: Maosong Sun
paper_authors._id: 65f3c3208e0f2653cdef4f7b
paper_authors.name: Shengnan Wang
paper_authors._id: 65f3c3208e0f2653cdef4f7c
paper_authors.name: Teng Su
paper_title: BurstAttention: An Efficient Distributed Attention Framework for
    Extremely Long Sequences
paper_summary: Effective attention modules have played a crucial role in the success of
  Transformer-based large language models (LLMs), but the quadratic time and
  memory complexities of these attention modules also pose a challenge when
  processing long sequences. One potential solution for the long sequence problem
  is to utilize distributed clusters to parallelize the computation of attention
  modules across multiple devices (e.g., GPUs). However, adopting a distributed
  approach inevitably introduces extra memory overheads to store local attention
  results and incurs additional communication costs to aggregate local results
  into global ones. In this paper, we propose a distributed attention framework
  named ``BurstAttention'' to optimize memory access and communication operations
  at both the global cluster and local device levels. In our experiments, we
  compare BurstAttention with other competitive distributed attention solutions
  for long sequence processing. The experimental results under different length
  settings demonstrate that BurstAttention offers significant advantages for
  processing long sequences compared with these competitive baselines, reducing
  40% communication overheads and achieving 2 X speedup during training 32K
  sequence length on 8 X A100.
paper_ai_keywords: attention modules Transformer-based large language models (LLMs) quadratic time and memory complexities distributed clusters parallelize the computation GPUs memory overheads communication costs local attention results global results distributed attention framework BurstAttention memory access communication operations long sequence processing competitive distributed attention solutions