FFN Fusion: Rethinking Sequential Computation in Large Language Models
We introduce FFN Fusion, an architectural optimization technique that reduces
sequential computation in large language models by identifying and exploiting
natural opportunities for parallelization. Our key insight is that sequences of
Feed-Forward Network (FFN) layers, particularly those remaining after the
removal of specific attention layers, can often be parallelized with minimal
accuracy impact. We develop a principled methodology for identifying and fusing
such sequences, transforming them into parallel operations that significantly
reduce inference latency while preserving model behavior. Applying these
techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base
(Ultra-253B-Base), an efficient and soon-to-be publicly available model that
achieves a 1.71X speedup in inference latency and 35X lower per-token cost
while maintaining strong performance across benchmarks. Through extensive
experiments on models from 49B to 253B parameters, we demonstrate that FFN
Fusion becomes increasingly effective at larger scales and can complement
existing optimization techniques like quantization and pruning. Most
intriguingly, we find that even full transformer blocks containing both
attention and FFN layers can sometimes be parallelized, suggesting new
directions for neural architecture design.
67e230fd4b9f234b60d06389
Akhiad Bercovich
admin_assigned
2025-03-25T09:30:49.482Z
66857bd849a4ed9de4c31936
/avatars/f6f016bf36fad5b29f30fbec6cde3e4d.svg
Akhiad Bercovich
user
abercovich
67e230fd4b9f234b60d0638a
Mohammad Dabbah
admin_assigned
2025-03-25T09:31:05.696Z
6756aa3741b39ab0d327de52
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VjjYWljIgPn9HEMDyKtft.png
Mohammad Dabbah
user
mdabbah-nvidia
67e230fd4b9f234b60d0638b
Omri Puny
admin_assigned
2025-03-25T09:31:13.161Z
6509a96c61c4bb4636fd0fd2
/avatars/8ffa9b4dd698469f7d70d4d9144aac82.svg
Omri Puny
user
omripuny
67e230fd4b9f234b60d0638c
Ido Galil
67e230fd4b9f234b60d0638d
Amnon Geifman
admin_assigned
2025-03-25T09:31:22.938Z
65006cac12c1442d993d6d51
/avatars/6700109303b902d453f3d8e2b45a103f.svg
Geifman
user
AmnonGeifman
67e230fd4b9f234b60d0638e
Yonatan Geifman
admin_assigned
2025-03-25T09:31:30.392Z
604bc69e0fe8ff3ec13d71cd
/avatars/fe4b14b24befdbed02eecb43a25c67f4.svg
Yonatan Geifman
user
geifmany
67e230fd4b9f234b60d0638f
Izhak Golan
67e230fd4b9f234b60d06390
Ehud Karpas
67e230fd4b9f234b60d06391
Itay Levy
admin_assigned
2025-03-25T09:31:50.928Z
668578fdd24e614fec97eac8
https://cdn-avatars.huggingface.co/v1/production/uploads/668578fdd24e614fec97eac8/n5xYnqo5nQbVX2tgaRfEi.jpeg
Itay Levy
user
itlevy
67e230fd4b9f234b60d06392
Zach Moshe
admin_assigned
2025-03-25T09:31:56.910Z
61ee58f1af500c0acfc4d8eb
https://cdn-avatars.huggingface.co/v1/production/uploads/1643010228464-noauth.png
Zach Moshe
user
zachmoshe
67e230fd4b9f234b60d06393
Najeeb Nabwani
admin_assigned
2025-03-25T09:32:02.613Z
63a16d5d5d09b819fee9a350
/avatars/d1a3fef0131688e92e272cbd80856fc3.svg
Najeeb Nabwani
user
NajeebDeci
67e230fd4b9f234b60d06394
Tomer Ronen
admin_assigned
2025-03-25T09:32:10.072Z
6671634f1820f293a9995b12
/avatars/50c8f7b4bfb00f2169b808f3c72c7686.svg
Tomer Ronen
user
tomer-nv
67e230fd4b9f234b60d06395
Itamar Schen
admin_assigned
2025-03-25T09:32:17.605Z
665f0a46f065b1d42806000d
/avatars/927f042a3c95c5846621e2a381c66bbf.svg
Itamar Schen
user
ischen-nvidia
67e230fd4b9f234b60d06396
Elad Segal
admin_assigned
2025-03-25T09:32:24.511Z
5f5b0efe10b2753d9000c888
https://cdn-avatars.huggingface.co/v1/production/uploads/1628140531144-5f5b0efe10b2753d9000c888.jpeg
Elad Segal
user
eladsegal
67e230fd4b9f234b60d06397
Ido Shahaf
admin_assigned
2025-03-25T09:32:30.927Z
666ef13c14f1c262feeb706c
/avatars/7dca59acf5e069d96bdbb98dace9199b.svg
Ido Shahaf
user
ishahaf
67e230fd4b9f234b60d06398
Oren Tropp
admin_assigned
2025-03-25T09:32:38.161Z
66b089f14ae4ae811218cdb6
/avatars/a50fe725922dfdbe0e731fade381b22e.svg
Oren Tropp
user
otropp
67e230fd4b9f234b60d06399
Ran Zilberstein
admin_assigned
2025-03-25T09:32:44.768Z
666027917c3f9c72113cc75c
/avatars/a276ebe8e2731b6a05e3c61c2ae0ddae.svg
Ran Zilberstein
user
RanZilberstein-Nvidia
67e230fd4b9f234b60d0639a
Ran El-Yaniv
admin_assigned
2025-03-25T09:32:56.726Z
65758349983403462a54ac06
/avatars/4f337c732f31bd748738c2717b50a99c.svg
Ran El-Yaniv
user
ranielyaniv
FFN Fusion: Rethinking Sequential Computation in Large Language Models
We introduce FFN Fusion, an architectural optimization technique that reduces
sequential computation in large language models by identifying and exploiting
natural opportunities for parallelization. Our key insight is that sequences of
Feed-Forward Network (FFN) layers, particularly those remaining after the
removal of specific attention layers, can often be parallelized with minimal
accuracy impact. We develop a principled methodology for identifying and fusing
such sequences, transforming them into parallel operations that significantly
reduce inference latency while preserving model behavior. Applying these
techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base
(Ultra-253B-Base), an efficient and soon-to-be publicly available model that
achieves a 1.71X speedup in inference latency and 35X lower per-token cost
while maintaining strong performance across benchmarks. Through extensive
experiments on models from 49B to 253B parameters, we demonstrate that FFN
Fusion becomes increasingly effective at larger scales and can complement
existing optimization techniques like quantization and pruning. Most
intriguingly, we find that even full transformer blocks containing both
attention and FFN layers can sometimes be parallelized, suggesting new
directions for neural architecture design.
FFN Fusion reduces sequential computation in large language models by parallelizing FFN layers, achieving significant speedup and cost reduction while maintaining performance.
FFN Fusion Feed-Forward Network inference latency parameter-efficient quantization pruning full transformer blocks parallelization