title: DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme
    Long Sequence Transformer Models
summary: Computation in a typical Transformer-based large language model (LLM) can be
  characterized by batch size, hidden dimension, number of layers, and sequence
  length. Until now, system works for accelerating LLM training have focused on
  the first three dimensions: data parallelism for batch size, tensor parallelism
  for hidden size and pipeline parallelism for model depth or layers. These
  widely studied forms of parallelism are not targeted or optimized for long
  sequence Transformer models. Given practical application needs for long
  sequence LLM, renewed attentions are being drawn to sequence parallelism.
  However, existing works in sequence parallelism are constrained by
  memory-communication inefficiency, limiting their scalability to long sequence
  large models. In this work, we introduce DeepSpeed-Ulysses, a novel, portable
  and effective methodology for enabling highly efficient and scalable LLM
  training with extremely long sequence length. DeepSpeed-Ulysses at its core
  partitions input data along the sequence dimension and employs an efficient
  all-to-all collective communication for attention computation. Theoretical
  communication analysis shows that whereas other methods incur communication
  overhead as sequence length increases, DeepSpeed-Ulysses maintains constant
  communication volume when sequence length and compute devices are increased
  proportionally. Furthermore, experimental evaluations show that
  DeepSpeed-Ulysses trains 2.5X faster with 4X longer sequence length than the
  existing method SOTA baseline.
paper_authors._id: 6513924a606fc2019243eb7f
paper_authors.name: Sam Ade Jacobs
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-02T07:00:29.319Z
paper_authors._id: 651736dd20b18e99b44dd5d5
paper_authors.avatarUrl: /avatars/c054eb2567e027408340a03a7ca0b29d.svg
paper_authors.fullname: Sam Ade Jacobs
paper_authors.type: user
paper_authors.user: samadejacobs
paper_authors._id: 6513924a606fc2019243eb80
paper_authors.name: Masahiro Tanaka
paper_authors._id: 6513924a606fc2019243eb81
paper_authors.name: Chengming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:31:09.240Z
paper_authors._id: 6480355ccc9e514b3b033a78
paper_authors.avatarUrl: /avatars/ff86bd27261c755c20546b400392f256.svg
paper_authors.fullname: Chengming Zhang
paper_authors.type: user
paper_authors.user: cc32
paper_authors._id: 6513924a606fc2019243eb82
paper_authors.name: Minjia Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:29:23.957Z
paper_authors._id: 6305b5e39d2531fabd195c5f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6305b5e39d2531fabd195c5f/oGS5leXVrB8Ct_BF96VkR.jpeg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Minjia
paper_authors._id: 6513924a606fc2019243eb83
paper_authors.name: Leon Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-27T15:21:16.142Z
paper_authors._id: 651442f5c75a3d4c44de4ac9
paper_authors.avatarUrl: /avatars/c60185389bf6379e4360cb54615922ed.svg
paper_authors.fullname: Shuaiwen Leon Song
paper_authors.type: user
paper_authors.user: leonangel991
paper_authors._id: 6513924a606fc2019243eb84
paper_authors.name: Samyam Rajbhandari
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-27T10:29:50.197Z
paper_authors._id: 63238e79d444e1d90888ea03
paper_authors.avatarUrl: /avatars/cd0accb5ddec4f2ae23c94da4094b5e5.svg
paper_authors.fullname: Samyam Rajbhandari
paper_authors.type: user
paper_authors.user: samyam
paper_authors._id: 6513924a606fc2019243eb85
paper_authors.name: Yuxiong He
paper_title: DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme
    Long Sequence Transformer Models
paper_summary: Computation in a typical Transformer-based large language model (LLM) can be
  characterized by batch size, hidden dimension, number of layers, and sequence
  length. Until now, system works for accelerating LLM training have focused on
  the first three dimensions: data parallelism for batch size, tensor parallelism
  for hidden size and pipeline parallelism for model depth or layers. These
  widely studied forms of parallelism are not targeted or optimized for long
  sequence Transformer models. Given practical application needs for long
  sequence LLM, renewed attentions are being drawn to sequence parallelism.
  However, existing works in sequence parallelism are constrained by
  memory-communication inefficiency, limiting their scalability to long sequence
  large models. In this work, we introduce DeepSpeed-Ulysses, a novel, portable
  and effective methodology for enabling highly efficient and scalable LLM
  training with extremely long sequence length. DeepSpeed-Ulysses at its core
  partitions input data along the sequence dimension and employs an efficient
  all-to-all collective communication for attention computation. Theoretical
  communication analysis shows that whereas other methods incur communication
  overhead as sequence length increases, DeepSpeed-Ulysses maintains constant
  communication volume when sequence length and compute devices are increased
  proportionally. Furthermore, experimental evaluations show that
  DeepSpeed-Ulysses trains 2.5X faster with 4X longer sequence length than the
  existing method SOTA baseline.
paper_ai_summary: DeepSpeed-Ulysses enables efficient and scalable training of Transformer-based LLMs with long sequences by partitioning input data along the sequence dimension and using all-to-all communication.
paper_ai_keywords: Transformer-based large language model LLM data parallelism tensor parallelism pipeline parallelism sequence parallelism memory-communication inefficiency DeepSpeed-Ulysses all-to-all collective communication attention computation communication analysis