title: Ultra-Long Sequence Distributed Transformer
summary: Transformer models trained on long sequences often achieve higher accuracy
  than short sequences. Unfortunately, conventional transformers struggle with
  long sequence training due to the overwhelming computation and memory
  requirements. Existing methods for long sequence training offer limited speedup
  and memory reduction, and may compromise accuracy. This paper presents a novel
  and efficient distributed training method, the Long Short-Sequence Transformer
  (LSS Transformer), for training transformer with long sequences. It distributes
  a long sequence into segments among GPUs, with each GPU computing a partial
  self-attention for its segment. Then, it uses a fused communication and a novel
  double gradient averaging technique to avoid the need to aggregate partial
  self-attention and minimize communication overhead. We evaluated the
  performance between LSS Transformer and the state-of-the-art Nvidia sequence
  parallelism on a Wikipedia enwik8 dataset. Results show that our proposed
  method lead to 5.6x faster and 10.2x more memory-efficient implementation
  compared to state-of-the-art sequence parallelism on 144 Nvidia V100 GPUs.
  Moreover, our algorithm scales to an extreme sequence length of 50,112 at 3,456
  GPUs, achieving 161% super-linear parallel efficiency and a throughput of 32
  petaflops.
paper_authors._id: 6549e7c19b86bd6b2c79712e
paper_authors.name: Xiao Wang
paper_authors._id: 6549e7c19b86bd6b2c79712f
paper_authors.name: Isaac Lyngaas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T16:25:19.754Z
paper_authors._id: 628cd80686fc004b14e3c7a7
paper_authors.avatarUrl: /avatars/316afa1d3fd02993fe246f002627b69a.svg
paper_authors.fullname: Isaac Lyngaas
paper_authors.type: user
paper_authors.user: irlyngaas
paper_authors._id: 6549e7c19b86bd6b2c797130
paper_authors.name: Aristeidis Tsaris
paper_authors._id: 6549e7c19b86bd6b2c797131
paper_authors.name: Peng Chen
paper_authors._id: 6549e7c19b86bd6b2c797132
paper_authors.name: Sajal Dash
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T16:25:50.740Z
paper_authors._id: 627587897128d87e9f928d16
paper_authors.avatarUrl: /avatars/fdc3e4524a5b71e9951a7ee4c3384e95.svg
paper_authors.fullname: Sajal Dash
paper_authors.type: user
paper_authors.user: sajaldash
paper_authors._id: 6549e7c19b86bd6b2c797133
paper_authors.name: Mayanka Chandra Shekar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T16:25:57.774Z
paper_authors._id: 6283d8b0d092918e5ed99223
paper_authors.avatarUrl: /avatars/6702c6a546de432b39dea7a4f1e3d706.svg
paper_authors.fullname: Mayanka Chandra Shekar
paper_authors.type: user
paper_authors.user: mayanka
paper_authors._id: 6549e7c19b86bd6b2c797134
paper_authors.name: Tao Luo
paper_authors._id: 6549e7c19b86bd6b2c797135
paper_authors.name: Hong-Jun Yoon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T16:26:49.006Z
paper_authors._id: 649edc1bd5065637167248ab
paper_authors.avatarUrl: /avatars/dc6f6299c5758861b5a0f5e3ec03a419.svg
paper_authors.fullname: Hong-Jun Yoon
paper_authors.type: user
paper_authors.user: hongjuny
paper_authors._id: 6549e7c19b86bd6b2c797136
paper_authors.name: Mohamed Wahib
paper_authors._id: 6549e7c19b86bd6b2c797137
paper_authors.name: John Gouley
paper_title: Ultra-Long Sequence Distributed Transformer
paper_summary: Transformer models trained on long sequences often achieve higher accuracy
  than short sequences. Unfortunately, conventional transformers struggle with
  long sequence training due to the overwhelming computation and memory
  requirements. Existing methods for long sequence training offer limited speedup
  and memory reduction, and may compromise accuracy. This paper presents a novel
  and efficient distributed training method, the Long Short-Sequence Transformer
  (LSS Transformer), for training transformer with long sequences. It distributes
  a long sequence into segments among GPUs, with each GPU computing a partial
  self-attention for its segment. Then, it uses a fused communication and a novel
  double gradient averaging technique to avoid the need to aggregate partial
  self-attention and minimize communication overhead. We evaluated the
  performance between LSS Transformer and the state-of-the-art Nvidia sequence
  parallelism on a Wikipedia enwik8 dataset. Results show that our proposed
  method lead to 5.6x faster and 10.2x more memory-efficient implementation
  compared to state-of-the-art sequence parallelism on 144 Nvidia V100 GPUs.
  Moreover, our algorithm scales to an extreme sequence length of 50,112 at 3,456
  GPUs, achieving 161% super-linear parallel efficiency and a throughput of 32
  petaflops.
paper_ai_keywords: Transformer models long sequences self-attention distributed training Long Short-Sequence Transformer (LSS Transformer) GPU fused communication double gradient averaging sequence parallelism Wikipedia enwik8 dataset super-linear parallel efficiency throughput petaflops