title: 2BP: 2-Stage Backpropagation
summary: As Deep Neural Networks (DNNs) grow in size and complexity, they often exceed
  the memory capacity of a single accelerator, necessitating the sharding of
  model parameters across multiple accelerators. Pipeline parallelism is a
  commonly used sharding strategy for training large DNNs. However, current
  implementations of pipeline parallelism are being unintentionally bottlenecked
  by the automatic differentiation tools provided by ML frameworks. This paper
  introduces 2-stage backpropagation (2BP). By splitting the backward propagation
  step into two separate stages, we can reduce idle compute time. We tested 2BP
  on various model architectures and pipelining schedules, achieving increases in
  throughput in all cases. Using 2BP, we were able to achieve a 1.70x increase in
  throughput compared to traditional methods when training a LLaMa-like
  transformer with 7 billion parameters across 4 GPUs.
paper_authors._id: 66569b0a76582ebbd3a0a604
paper_authors.name: Christopher Rae
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-29T08:23:51.603Z
paper_authors._id: 62ec07852780dfe4e2debd7b
paper_authors.avatarUrl: /avatars/6391e82a07492b2d642c37d08c120013.svg
paper_authors.fullname: Christopher Rae
paper_authors.type: user
paper_authors.user: mayfieldmob
paper_authors._id: 66569b0a76582ebbd3a0a605
paper_authors.name: Joseph K. L. Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:25:52.519Z
paper_authors._id: 65d9d0ae0e6ad24551caf7de
paper_authors.avatarUrl: /avatars/baf7e8b25631bc9db0f907e474cf40cd.svg
paper_authors.fullname: Joseph Lee
paper_authors.type: user
paper_authors.user: josephleekl
paper_authors._id: 66569b0a76582ebbd3a0a606
paper_authors.name: James Richings
paper_title: 2BP: 2-Stage Backpropagation
paper_summary: As Deep Neural Networks (DNNs) grow in size and complexity, they often exceed
  the memory capacity of a single accelerator, necessitating the sharding of
  model parameters across multiple accelerators. Pipeline parallelism is a
  commonly used sharding strategy for training large DNNs. However, current
  implementations of pipeline parallelism are being unintentionally bottlenecked
  by the automatic differentiation tools provided by ML frameworks. This paper
  introduces 2-stage backpropagation (2BP). By splitting the backward propagation
  step into two separate stages, we can reduce idle compute time. We tested 2BP
  on various model architectures and pipelining schedules, achieving increases in
  throughput in all cases. Using 2BP, we were able to achieve a 1.70x increase in
  throughput compared to traditional methods when training a LLaMa-like
  transformer with 7 billion parameters across 4 GPUs.
paper_ai_keywords: Deep Neural Networks (DNNs) model parameters accelerators pipeline parallelism backward propagation idle compute time throughput LLaMa-like transformer