title: Zero Bubble Pipeline Parallelism
summary: Pipeline parallelism is one of the key components for large-scale distributed
  training, yet its efficiency suffers from pipeline bubbles which were deemed
  inevitable. In this work, we introduce a scheduling strategy that, to our
  knowledge, is the first to successfully achieve zero pipeline bubbles under
  synchronous training semantics. The key idea behind this improvement is to
  split the backward computation into two parts, one that computes gradient for
  the input and another that computes for the parameters. Based on this idea, we
  handcraft novel pipeline schedules that significantly outperform the baseline
  methods. We further develop an algorithm that automatically finds an optimal
  schedule based on specific model configuration and memory limit. Additionally,
  to truly achieve zero bubble, we introduce a novel technique to bypass
  synchronizations during the optimizer step. Experimental evaluations show that
  our method outperforms the 1F1B schedule up to 23% in throughput under a
  similar memory limit. This number can be further pushed to 31% when the memory
  constraint is relaxed. We believe our results mark a major step forward in
  harnessing the true potential of pipeline parallelism. We open sourced our
  implementation based on the popular Megatron-LM repository on
  https://github.com/sail-sg/zero-bubble-pipeline-parallelism.
paper_authors._id: 65addffdb68db4f26e0eca49
paper_authors.name: Penghui Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:28:51.296Z
paper_authors._id: 63885f1d0bebb233d8ad6e5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
paper_authors.fullname: Penghui Qi
paper_authors.type: user
paper_authors.user: QPHutu
paper_authors._id: 65addffdb68db4f26e0eca4a
paper_authors.name: Xinyi Wan
paper_authors._id: 65addffdb68db4f26e0eca4b
paper_authors.name: Guangxing Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-22T11:39:27.407Z
paper_authors._id: 6429a8688852afdf89be7b50
paper_authors.avatarUrl: /avatars/093a68dda97904271700359b3f6684ef.svg
paper_authors.fullname: huangguangxing
paper_authors.type: user
paper_authors.user: huangguangxing
paper_authors._id: 65addffdb68db4f26e0eca4c
paper_authors.name: Min Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T12:00:44.739Z
paper_authors._id: 6565d5fd656151d58b1d9afb
paper_authors.avatarUrl: /avatars/0c416ea74347c6b5bd27f49aec309c48.svg
paper_authors.fullname: Min Lin
paper_authors.type: user
paper_authors.user: mavenlin
paper_title: Zero Bubble Pipeline Parallelism
paper_summary: Pipeline parallelism is one of the key components for large-scale distributed
  training, yet its efficiency suffers from pipeline bubbles which were deemed
  inevitable. In this work, we introduce a scheduling strategy that, to our
  knowledge, is the first to successfully achieve zero pipeline bubbles under
  synchronous training semantics. The key idea behind this improvement is to
  split the backward computation into two parts, one that computes gradient for
  the input and another that computes for the parameters. Based on this idea, we
  handcraft novel pipeline schedules that significantly outperform the baseline
  methods. We further develop an algorithm that automatically finds an optimal
  schedule based on specific model configuration and memory limit. Additionally,
  to truly achieve zero bubble, we introduce a novel technique to bypass
  synchronizations during the optimizer step. Experimental evaluations show that
  our method outperforms the 1F1B schedule up to 23% in throughput under a
  similar memory limit. This number can be further pushed to 31% when the memory
  constraint is relaxed. We believe our results mark a major step forward in
  harnessing the true potential of pipeline parallelism. We open sourced our
  implementation based on the popular Megatron-LM repository on
  https://github.com/sail-sg/zero-bubble-pipeline-parallelism.
paper_ai_keywords: pipeline parallelism synchronous training semantics pipeline bubbles backward computation gradient computation handcrafted pipeline schedules automatic scheduling algorithm optimizer step synchronization bypass Megatron-LM repository