title: Blockwise Parallel Transformer for Long Context Large Models
summary: Transformers have emerged as the cornerstone of state-of-the-art natural
  language processing models, showcasing exceptional performance across a wide
  range of AI applications. However, the memory demands posed by the
  self-attention mechanism and the large feedforward network in Transformers
  limit their ability to handle long sequences, thereby creating challenges for
  tasks involving multiple long sequences or long-term dependencies. We present a
  distinct approach, Blockwise Parallel Transformer (BPT), that leverages
  blockwise computation of self-attention and feedforward network fusion to
  minimize memory costs. By processing longer input sequences while maintaining
  memory efficiency, BPT enables training sequences up to 32 times longer than
  vanilla Transformers and 2 to 4 times longer than previous memory-efficient
  methods. Extensive experiments on language modeling and reinforcement learning
  tasks demonstrate the effectiveness of BPT in reducing memory requirements and
  improving performance.
paper_authors._id: 6477f962c21496284db746f4
paper_authors.name: Hao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-07T12:39:28.395Z
paper_authors._id: 62aa53fc5df2251f7df798c6
paper_authors.avatarUrl: /avatars/1c7828ce861c30ddd0afceff355a92ec.svg
paper_authors.fullname: Hao
paper_authors.type: user
paper_authors.user: haoliu
paper_authors._id: 6477f962c21496284db746f5
paper_authors.name: Pieter Abbeel
paper_title: Blockwise Parallel Transformer for Long Context Large Models
paper_summary: Transformers have emerged as the cornerstone of state-of-the-art natural
  language processing models, showcasing exceptional performance across a wide
  range of AI applications. However, the memory demands posed by the
  self-attention mechanism and the large feedforward network in Transformers
  limit their ability to handle long sequences, thereby creating challenges for
  tasks involving multiple long sequences or long-term dependencies. We present a
  distinct approach, Blockwise Parallel Transformer (BPT), that leverages
  blockwise computation of self-attention and feedforward network fusion to
  minimize memory costs. By processing longer input sequences while maintaining
  memory efficiency, BPT enables training sequences up to 32 times longer than
  vanilla Transformers and 2 to 4 times longer than previous memory-efficient
  methods. Extensive experiments on language modeling and reinforcement learning
  tasks demonstrate the effectiveness of BPT in reducing memory requirements and
  improving performance.
paper_ai_summary: Blockwise Parallel Transformer (BPT) enhances memory efficiency and handles longer sequences by optimizing self-attention and feedforward network computations.
paper_ai_keywords: Transformers self-attention mechanism feedforward network Blockwise Parallel Transformer BPT memory-efficient