title: Brainformers: Trading Simplicity for Efficiency
summary: Transformers are central to recent successes in natural language processing
  and computer vision. Transformers have a mostly uniform backbone where layers
  alternate between feed-forward and self-attention in order to build a deep
  network. Here we investigate this design choice and find that more complex
  blocks that have different permutations of layer primitives can be more
  efficient. Using this insight, we develop a complex block, named Brainformer,
  that consists of a diverse sets of layers such as sparsely gated feed-forward
  layers, dense feed-forward layers, attention layers, and various forms of layer
  normalization and activation functions. Brainformer consistently outperforms
  the state-of-the-art dense and sparse Transformers, in terms of both quality
  and efficiency. A Brainformer model with 8 billion activated parameters per
  token demonstrates 2x faster training convergence and 5x faster step time
  compared to its GLaM counterpart. In downstream task evaluation, Brainformer
  also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM
  with a similar number of activated parameters. Finally, Brainformer largely
  outperforms a Primer dense model derived with NAS with similar computation per
  token on fewshot evaluations.
paper_authors._id: 647975a0a68454566356af42
paper_authors.name: Yanqi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-05T10:07:51.287Z
paper_authors._id: 643c3c0a9f5d314db2dcba18
paper_authors.avatarUrl: /avatars/6e6599c558b88c97144008ada1abb5f7.svg
paper_authors.fullname: Yanqi Zhou
paper_authors.type: user
paper_authors.user: yanqiz
paper_authors._id: 647975a0a68454566356af43
paper_authors.name: Nan Du
paper_authors._id: 647975a0a68454566356af44
paper_authors.name: Yanping Huang
paper_authors._id: 647975a0a68454566356af45
paper_authors.name: Daiyi Peng
paper_authors._id: 647975a0a68454566356af46
paper_authors.name: Chang Lan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T12:06:12.854Z
paper_authors._id: 63003b224f0095df258f3015
paper_authors.avatarUrl: /avatars/b4c3957df138b2a4fd296e13b131a5f2.svg
paper_authors.fullname: Chang Lan
paper_authors.type: user
paper_authors.user: changlan
paper_authors._id: 647975a0a68454566356af47
paper_authors.name: Da Huang
paper_authors._id: 647975a0a68454566356af48
paper_authors.name: Siamak Shakeri
paper_authors._id: 647975a0a68454566356af49
paper_authors.name: David So
paper_authors._id: 647975a0a68454566356af4a
paper_authors.name: Andrew Dai
paper_authors._id: 647975a0a68454566356af4b
paper_authors.name: Yifeng Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-05T12:07:17.432Z
paper_authors._id: 6344a8050f69ad8aa61cb663
paper_authors.avatarUrl: /avatars/ed637f510d18ccd1f4bdd6861d5c858e.svg
paper_authors.fullname: Yifeng Lu
paper_authors.type: user
paper_authors.user: yifenglu
paper_authors._id: 647975a0a68454566356af4c
paper_authors.name: Zhifeng Chen
paper_authors._id: 647975a0a68454566356af4d
paper_authors.name: Quoc Le
paper_authors._id: 647975a0a68454566356af4e
paper_authors.name: Claire Cui
paper_authors._id: 647975a0a68454566356af4f
paper_authors.name: James Laundon
paper_authors._id: 647975a0a68454566356af50
paper_authors.name: Jeff Dean
paper_title: Brainformers: Trading Simplicity for Efficiency
paper_summary: Transformers are central to recent successes in natural language processing
  and computer vision. Transformers have a mostly uniform backbone where layers
  alternate between feed-forward and self-attention in order to build a deep
  network. Here we investigate this design choice and find that more complex
  blocks that have different permutations of layer primitives can be more
  efficient. Using this insight, we develop a complex block, named Brainformer,
  that consists of a diverse sets of layers such as sparsely gated feed-forward
  layers, dense feed-forward layers, attention layers, and various forms of layer
  normalization and activation functions. Brainformer consistently outperforms
  the state-of-the-art dense and sparse Transformers, in terms of both quality
  and efficiency. A Brainformer model with 8 billion activated parameters per
  token demonstrates 2x faster training convergence and 5x faster step time
  compared to its GLaM counterpart. In downstream task evaluation, Brainformer
  also demonstrates a 3% higher SuperGLUE score with fine-tuning compared to GLaM
  with a similar number of activated parameters. Finally, Brainformer largely
  outperforms a Primer dense model derived with NAS with similar computation per
  token on fewshot evaluations.
paper_ai_keywords: feed-forward self-attention Brainformer sparsely-gated feed-forward layers dense feed-forward layers attention layers layer normalization activation functions dense and sparse Transformers GLaM SuperGLUE few-shot evaluations NAS (Neural Architecture Search)