title: PipeOffload: Improving Scalability of Pipeline Parallelism with Memory
    Optimization
summary: Pipeline parallelism (PP) is widely used for training large language models
  (LLMs), yet its scalability is often constrained by high activation memory
  consumption as the number of in-flight microbatches grows with the degree of
  PP. In this paper, we focus on addressing this challenge by leveraging the
  under-explored memory offload strategy in PP. With empirical study, we discover
  that in the majority of standard configurations, at least half, and potentially
  all, of the activations can be offloaded with negligible overhead. In the cases
  where full overload is not possible, we introduce a novel selective offload
  strategy that decreases peak activation memory in a better-than-linear manner.
  Furthermore, we integrate memory offload with other techniques to jointly
  consider overall throughput and memory limitation. Our experiments proves that
  the per-device activation memory effectively reduces with the total number of
  stages, making PP a stronger alternative than TP, offering up to a 19\%
  acceleration with even lower memory consumption. The implementation is
  open-sourced at
  https://github.com/sail-sg/zero-bubble-pipeline-parallelism{this url}.
paper_authors._id: 67c7b5900b05ab9c7e805433
paper_authors.name: Xinyi Wan
paper_authors._id: 67c7b5900b05ab9c7e805434
paper_authors.name: Penghui Qi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:12:19.621Z
paper_authors._id: 63885f1d0bebb233d8ad6e5b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669881620925-noauth.jpeg
paper_authors.fullname: Penghui Qi
paper_authors.type: user
paper_authors.user: QPHutu
paper_authors._id: 67c7b5900b05ab9c7e805435
paper_authors.name: Guangxing Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:39:20.500Z
paper_authors._id: 67c7b8c95d393f85737562e5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/KMmEVbXppNkfj1T9UOD7o.png
paper_authors.fullname: Guangxing Huang
paper_authors.type: user
paper_authors.user: huanggx-sea
paper_authors._id: 67c7b5900b05ab9c7e805436
paper_authors.name: Jialin Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:39:29.789Z
paper_authors._id: 658f147c50d39af7f4cd889d
paper_authors.avatarUrl: /avatars/93bb33e975ab97c1f4179f20d384438f.svg
paper_authors.fullname: Jialin Li
paper_authors.type: user
paper_authors.user: JialinLi
paper_authors._id: 67c7b5900b05ab9c7e805437
paper_authors.name: Min Lin
paper_title: PipeOffload: Improving Scalability of Pipeline Parallelism with Memory
    Optimization
paper_summary: Pipeline parallelism (PP) is widely used for training large language models
  (LLMs), yet its scalability is often constrained by high activation memory
  consumption as the number of in-flight microbatches grows with the degree of
  PP. In this paper, we focus on addressing this challenge by leveraging the
  under-explored memory offload strategy in PP. With empirical study, we discover
  that in the majority of standard configurations, at least half, and potentially
  all, of the activations can be offloaded with negligible overhead. In the cases
  where full overload is not possible, we introduce a novel selective offload
  strategy that decreases peak activation memory in a better-than-linear manner.
  Furthermore, we integrate memory offload with other techniques to jointly
  consider overall throughput and memory limitation. Our experiments proves that
  the per-device activation memory effectively reduces with the total number of
  stages, making PP a stronger alternative than TP, offering up to a 19\%
  acceleration with even lower memory consumption. The implementation is
  open-sourced at
  https://github.com/sail-sg/zero-bubble-pipeline-parallelism{this url}.
paper_ai_keywords: pipeline parallelism (PP) large language models (LLMs) activation memory microbatches memory offload selective offload per-device activation memory throughput memory limitation tensor parallelism (TP)