title: One Wide Feedforward is All You Need
summary: The Transformer architecture has two main non-embedding components: Attention
  and the Feed Forward Network (FFN). Attention captures interdependencies
  between words regardless of their position, while the FFN non-linearly
  transforms each input token independently. In this work we explore the role of
  the FFN, and find that despite taking up a significant fraction of the model's
  parameters, it is highly redundant. Concretely, we are able to substantially
  reduce the number of parameters with only a modest drop in accuracy by removing
  the FFN on the decoder layers and sharing a single FFN across the encoder.
  Finally we scale this architecture back to its original size by increasing the
  hidden dimension of the shared FFN, achieving substantial gains in both
  accuracy and latency with respect to the original Transformer Big.
paper_authors._id: 64f832721435cc7faa41680f
paper_authors.name: Telmo Pessoa Pires
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-06T08:53:45.354Z
paper_authors._id: 64e4b9bda236a0de4f773f14
paper_authors.avatarUrl: /avatars/755bc4ddd3970b0bbae72b4f5b06786b.svg
paper_authors.fullname: Telmo
paper_authors.type: user
paper_authors.user: telmop
paper_authors._id: 64f832721435cc7faa416810
paper_authors.name: António V. Lopes
paper_authors._id: 64f832721435cc7faa416811
paper_authors.name: Yannick Assogba
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-02T09:01:50.222Z
paper_authors._id: 65bbd15c719492167d5ff857
paper_authors.avatarUrl: /avatars/1fca4b9057bdae8a9d589e6fc970e8fa.svg
paper_authors.fullname: Yannick Assogba
paper_authors.type: user
paper_authors.user: yassogba
paper_authors._id: 64f832721435cc7faa416812
paper_authors.name: Hendra Setiawan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-06T15:24:48.568Z
paper_authors._id: 63d3328ac20657d2434dbb0f
paper_authors.avatarUrl: /avatars/a05313a1cf96befe65782439c8c486b8.svg
paper_authors.fullname: Hendra Setiawan
paper_authors.type: user
paper_authors.user: hndrstwn
paper_title: One Wide Feedforward is All You Need
paper_summary: The Transformer architecture has two main non-embedding components: Attention
  and the Feed Forward Network (FFN). Attention captures interdependencies
  between words regardless of their position, while the FFN non-linearly
  transforms each input token independently. In this work we explore the role of
  the FFN, and find that despite taking up a significant fraction of the model's
  parameters, it is highly redundant. Concretely, we are able to substantially
  reduce the number of parameters with only a modest drop in accuracy by removing
  the FFN on the decoder layers and sharing a single FFN across the encoder.
  Finally we scale this architecture back to its original size by increasing the
  hidden dimension of the shared FFN, achieving substantial gains in both
  accuracy and latency with respect to the original Transformer Big.
paper_ai_keywords: Transformer Attention Feed Forward Network (FFN) parameters redundancy decoding layers hidden dimension performance latency Transformer Big