One Wide Feedforward is All You Need
The Transformer architecture has two main non-embedding components: Attention
and the Feed Forward Network (FFN). Attention captures interdependencies
between words regardless of their position, while the FFN non-linearly
transforms each input token independently. In this work we explore the role of
the FFN, and find that despite taking up a significant fraction of the model's
parameters, it is highly redundant. Concretely, we are able to substantially
reduce the number of parameters with only a modest drop in accuracy by removing
the FFN on the decoder layers and sharing a single FFN across the encoder.
Finally we scale this architecture back to its original size by increasing the
hidden dimension of the shared FFN, achieving substantial gains in both
accuracy and latency with respect to the original Transformer Big.
64f832721435cc7faa41680f
Telmo Pessoa Pires
extracted_confirmed
2023-09-06T08:53:45.354Z
64e4b9bda236a0de4f773f14
/avatars/755bc4ddd3970b0bbae72b4f5b06786b.svg
Telmo
user
telmop
64f832721435cc7faa416810
António V. Lopes
64f832721435cc7faa416811
Yannick Assogba
claimed_verified
2024-02-02T09:01:50.222Z
65bbd15c719492167d5ff857
/avatars/1fca4b9057bdae8a9d589e6fc970e8fa.svg
Yannick Assogba
user
yassogba
64f832721435cc7faa416812
Hendra Setiawan
claimed_verified
2023-09-06T15:24:48.568Z
63d3328ac20657d2434dbb0f
/avatars/a05313a1cf96befe65782439c8c486b8.svg
Hendra Setiawan
user
hndrstwn
One Wide Feedforward is All You Need
The Transformer architecture has two main non-embedding components: Attention
and the Feed Forward Network (FFN). Attention captures interdependencies
between words regardless of their position, while the FFN non-linearly
transforms each input token independently. In this work we explore the role of
the FFN, and find that despite taking up a significant fraction of the model's
parameters, it is highly redundant. Concretely, we are able to substantially
reduce the number of parameters with only a modest drop in accuracy by removing
the FFN on the decoder layers and sharing a single FFN across the encoder.
Finally we scale this architecture back to its original size by increasing the
hidden dimension of the shared FFN, achieving substantial gains in both
accuracy and latency with respect to the original Transformer Big.
The study shows that the Feed Forward Network in Transformers is highly redundant and its reduction or sharing within the model can lead to improved accuracy and latency without significant loss of performance.
Transformer architecture Attention Feed Forward Network (FFN) parameter redundancy hidden dimension