title: FFN Fusion: Rethinking Sequential Computation in Large Language Models
summary: We introduce FFN Fusion, an architectural optimization technique that reduces
  sequential computation in large language models by identifying and exploiting
  natural opportunities for parallelization. Our key insight is that sequences of
  Feed-Forward Network (FFN) layers, particularly those remaining after the
  removal of specific attention layers, can often be parallelized with minimal
  accuracy impact. We develop a principled methodology for identifying and fusing
  such sequences, transforming them into parallel operations that significantly
  reduce inference latency while preserving model behavior. Applying these
  techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base
  (Ultra-253B-Base), an efficient and soon-to-be publicly available model that
  achieves a 1.71X speedup in inference latency and 35X lower per-token cost
  while maintaining strong performance across benchmarks. Through extensive
  experiments on models from 49B to 253B parameters, we demonstrate that FFN
  Fusion becomes increasingly effective at larger scales and can complement
  existing optimization techniques like quantization and pruning. Most
  intriguingly, we find that even full transformer blocks containing both
  attention and FFN layers can sometimes be parallelized, suggesting new
  directions for neural architecture design.
paper_authors._id: 67e230fd4b9f234b60d06389
paper_authors.name: Akhiad Bercovich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:30:49.482Z
paper_authors._id: 66857bd849a4ed9de4c31936
paper_authors.avatarUrl: /avatars/f6f016bf36fad5b29f30fbec6cde3e4d.svg
paper_authors.fullname: Akhiad Bercovich
paper_authors.type: user
paper_authors.user: abercovich
paper_authors._id: 67e230fd4b9f234b60d0638a
paper_authors.name: Mohammad Dabbah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:31:05.696Z
paper_authors._id: 6756aa3741b39ab0d327de52
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/VjjYWljIgPn9HEMDyKtft.png
paper_authors.fullname: Mohammad Dabbah
paper_authors.type: user
paper_authors.user: mdabbah-nvidia
paper_authors._id: 67e230fd4b9f234b60d0638b
paper_authors.name: Omri Puny
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:31:13.161Z
paper_authors._id: 6509a96c61c4bb4636fd0fd2
paper_authors.avatarUrl: /avatars/8ffa9b4dd698469f7d70d4d9144aac82.svg
paper_authors.fullname: Omri Puny
paper_authors.type: user
paper_authors.user: omripuny
paper_authors._id: 67e230fd4b9f234b60d0638c
paper_authors.name: Ido Galil
paper_authors._id: 67e230fd4b9f234b60d0638d
paper_authors.name: Amnon Geifman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:31:22.938Z
paper_authors._id: 65006cac12c1442d993d6d51
paper_authors.avatarUrl: /avatars/6700109303b902d453f3d8e2b45a103f.svg
paper_authors.fullname: Geifman
paper_authors.type: user
paper_authors.user: AmnonGeifman
paper_authors._id: 67e230fd4b9f234b60d0638e
paper_authors.name: Yonatan Geifman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:31:30.392Z
paper_authors._id: 604bc69e0fe8ff3ec13d71cd
paper_authors.avatarUrl: /avatars/fe4b14b24befdbed02eecb43a25c67f4.svg
paper_authors.fullname: Yonatan Geifman
paper_authors.type: user
paper_authors.user: geifmany
paper_authors._id: 67e230fd4b9f234b60d0638f
paper_authors.name: Izhak Golan
paper_authors._id: 67e230fd4b9f234b60d06390
paper_authors.name: Ehud Karpas
paper_authors._id: 67e230fd4b9f234b60d06391
paper_authors.name: Itay Levy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:31:50.928Z
paper_authors._id: 668578fdd24e614fec97eac8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/668578fdd24e614fec97eac8/n5xYnqo5nQbVX2tgaRfEi.jpeg
paper_authors.fullname: Itay Levy
paper_authors.type: user
paper_authors.user: itlevy
paper_authors._id: 67e230fd4b9f234b60d06392
paper_authors.name: Zach Moshe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:31:56.910Z
paper_authors._id: 61ee58f1af500c0acfc4d8eb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1643010228464-noauth.png
paper_authors.fullname: Zach Moshe
paper_authors.type: user
paper_authors.user: zachmoshe
paper_authors._id: 67e230fd4b9f234b60d06393
paper_authors.name: Najeeb Nabwani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:02.613Z
paper_authors._id: 63a16d5d5d09b819fee9a350
paper_authors.avatarUrl: /avatars/d1a3fef0131688e92e272cbd80856fc3.svg
paper_authors.fullname: Najeeb Nabwani
paper_authors.type: user
paper_authors.user: NajeebDeci
paper_authors._id: 67e230fd4b9f234b60d06394
paper_authors.name: Tomer Ronen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:10.072Z
paper_authors._id: 6671634f1820f293a9995b12
paper_authors.avatarUrl: /avatars/50c8f7b4bfb00f2169b808f3c72c7686.svg
paper_authors.fullname: Tomer Ronen
paper_authors.type: user
paper_authors.user: tomer-nv
paper_authors._id: 67e230fd4b9f234b60d06395
paper_authors.name: Itamar Schen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:17.605Z
paper_authors._id: 665f0a46f065b1d42806000d
paper_authors.avatarUrl: /avatars/927f042a3c95c5846621e2a381c66bbf.svg
paper_authors.fullname: Itamar Schen
paper_authors.type: user
paper_authors.user: ischen-nvidia
paper_authors._id: 67e230fd4b9f234b60d06396
paper_authors.name: Elad Segal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:24.511Z
paper_authors._id: 5f5b0efe10b2753d9000c888
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628140531144-5f5b0efe10b2753d9000c888.jpeg
paper_authors.fullname: Elad Segal
paper_authors.type: user
paper_authors.user: eladsegal
paper_authors._id: 67e230fd4b9f234b60d06397
paper_authors.name: Ido Shahaf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:30.927Z
paper_authors._id: 666ef13c14f1c262feeb706c
paper_authors.avatarUrl: /avatars/7dca59acf5e069d96bdbb98dace9199b.svg
paper_authors.fullname: Ido Shahaf
paper_authors.type: user
paper_authors.user: ishahaf
paper_authors._id: 67e230fd4b9f234b60d06398
paper_authors.name: Oren Tropp
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:38.161Z
paper_authors._id: 66b089f14ae4ae811218cdb6
paper_authors.avatarUrl: /avatars/a50fe725922dfdbe0e731fade381b22e.svg
paper_authors.fullname: Oren Tropp
paper_authors.type: user
paper_authors.user: otropp
paper_authors._id: 67e230fd4b9f234b60d06399
paper_authors.name: Ran Zilberstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:44.768Z
paper_authors._id: 666027917c3f9c72113cc75c
paper_authors.avatarUrl: /avatars/a276ebe8e2731b6a05e3c61c2ae0ddae.svg
paper_authors.fullname: Ran Zilberstein
paper_authors.type: user
paper_authors.user: RanZilberstein-Nvidia
paper_authors._id: 67e230fd4b9f234b60d0639a
paper_authors.name: Ran El-Yaniv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-25T09:32:56.726Z
paper_authors._id: 65758349983403462a54ac06
paper_authors.avatarUrl: /avatars/4f337c732f31bd748738c2717b50a99c.svg
paper_authors.fullname: Ran El-Yaniv
paper_authors.type: user
paper_authors.user: ranielyaniv
paper_title: FFN Fusion: Rethinking Sequential Computation in Large Language Models
paper_summary: We introduce FFN Fusion, an architectural optimization technique that reduces
  sequential computation in large language models by identifying and exploiting
  natural opportunities for parallelization. Our key insight is that sequences of
  Feed-Forward Network (FFN) layers, particularly those remaining after the
  removal of specific attention layers, can often be parallelized with minimal
  accuracy impact. We develop a principled methodology for identifying and fusing
  such sequences, transforming them into parallel operations that significantly
  reduce inference latency while preserving model behavior. Applying these
  techniques to Llama-3.1-405B-Instruct, we create Llama-Nemotron-Ultra-253B-Base
  (Ultra-253B-Base), an efficient and soon-to-be publicly available model that
  achieves a 1.71X speedup in inference latency and 35X lower per-token cost
  while maintaining strong performance across benchmarks. Through extensive
  experiments on models from 49B to 253B parameters, we demonstrate that FFN
  Fusion becomes increasingly effective at larger scales and can complement
  existing optimization techniques like quantization and pruning. Most
  intriguingly, we find that even full transformer blocks containing both
  attention and FFN layers can sometimes be parallelized, suggesting new
  directions for neural architecture design.
paper_ai_summary: FFN Fusion reduces sequential computation in large language models by parallelizing FFN layers, achieving significant speedup and cost reduction while maintaining performance.
paper_ai_keywords: FFN Fusion Feed-Forward Network inference latency parameter-efficient quantization pruning full transformer blocks parallelization