Blockwise Parallel Transformer for Long Context Large Models
Transformers have emerged as the cornerstone of state-of-the-art natural
language processing models, showcasing exceptional performance across a wide
range of AI applications. However, the memory demands posed by the
self-attention mechanism and the large feedforward network in Transformers
limit their ability to handle long sequences, thereby creating challenges for
tasks involving multiple long sequences or long-term dependencies. We present a
distinct approach, Blockwise Parallel Transformer (BPT), that leverages
blockwise computation of self-attention and feedforward network fusion to
minimize memory costs. By processing longer input sequences while maintaining
memory efficiency, BPT enables training sequences up to 32 times longer than
vanilla Transformers and 2 to 4 times longer than previous memory-efficient
methods. Extensive experiments on language modeling and reinforcement learning
tasks demonstrate the effectiveness of BPT in reducing memory requirements and
improving performance.
6477f962c21496284db746f4
Hao Liu
claimed_verified
2023-08-07T12:39:28.395Z
62aa53fc5df2251f7df798c6
/avatars/1c7828ce861c30ddd0afceff355a92ec.svg
Hao
user
haoliu
6477f962c21496284db746f5
Pieter Abbeel
Blockwise Parallel Transformer for Long Context Large Models
Transformers have emerged as the cornerstone of state-of-the-art natural
language processing models, showcasing exceptional performance across a wide
range of AI applications. However, the memory demands posed by the
self-attention mechanism and the large feedforward network in Transformers
limit their ability to handle long sequences, thereby creating challenges for
tasks involving multiple long sequences or long-term dependencies. We present a
distinct approach, Blockwise Parallel Transformer (BPT), that leverages
blockwise computation of self-attention and feedforward network fusion to
minimize memory costs. By processing longer input sequences while maintaining
memory efficiency, BPT enables training sequences up to 32 times longer than
vanilla Transformers and 2 to 4 times longer than previous memory-efficient
methods. Extensive experiments on language modeling and reinforcement learning
tasks demonstrate the effectiveness of BPT in reducing memory requirements and
improving performance.
Blockwise Parallel Transformer (BPT) enhances memory efficiency and handles longer sequences by optimizing self-attention and feedforward network computations.
Transformers self-attention mechanism feedforward network Blockwise Parallel Transformer BPT memory-efficient