title: Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
summary: Large language models (LLMs) have demonstrated remarkable capabilities, but
  their adoption is limited by high computational costs during inference. While
  increasing parameter counts enhances accuracy, it also widens the gap between
  state-of-the-art capabilities and practical deployability. We present Puzzle, a
  framework to accelerate LLM inference on specific hardware while preserving
  their capabilities. Through an innovative application of neural architecture
  search (NAS) at an unprecedented scale, Puzzle systematically optimizes models
  with tens of billions of parameters under hardware constraints. Our approach
  utilizes blockwise local knowledge distillation (BLD) for parallel architecture
  exploration and employs mixed-integer programming for precise constraint
  optimization.
    We demonstrate the real-world impact of our framework through
  Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B), a publicly available model
  derived from Llama-3.1-70B-Instruct. Nemotron-51B achieves a 2.17x inference
  throughput speedup, fitting on a single NVIDIA H100 GPU while preserving 98.4%
  of the original model's capabilities. Nemotron-51B currently stands as the most
  accurate language model capable of inference on a single GPU with large batch
  sizes. Remarkably, this transformation required just 45B training tokens,
  compared to over 15T tokens used for the 70B model it was derived from. This
  establishes a new paradigm where powerful models can be optimized for efficient
  deployment with only negligible compromise of their capabilities, demonstrating
  that inference performance, not parameter count alone, should guide model
  selection. With the release of Nemotron-51B and the presentation of the Puzzle
  framework, we provide practitioners immediate access to state-of-the-art
  language modeling capabilities at significantly reduced computational costs.
paper_authors._id: 674d3de96ab71cc666cce97e
paper_authors.name: Akhiad Bercovich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:21:17.415Z
paper_authors._id: 66857bd849a4ed9de4c31936
paper_authors.avatarUrl: /avatars/f6f016bf36fad5b29f30fbec6cde3e4d.svg
paper_authors.fullname: Akhiad Bercovich
paper_authors.type: user
paper_authors.user: abercovich
paper_authors._id: 674d3de96ab71cc666cce97f
paper_authors.name: Tomer Ronen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:21:40.810Z
paper_authors._id: 6671634f1820f293a9995b12
paper_authors.avatarUrl: /avatars/50c8f7b4bfb00f2169b808f3c72c7686.svg
paper_authors.fullname: Tomer Ronen
paper_authors.type: user
paper_authors.user: tomer-nv
paper_authors._id: 674d3de96ab71cc666cce980
paper_authors.name: Talor Abramovich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:21:55.666Z
paper_authors._id: 668e7a379ab7631fcd434ee7
paper_authors.avatarUrl: /avatars/68ddbc5e35125a779655e9033d24afe5.svg
paper_authors.fullname: Talor Abramovich
paper_authors.type: user
paper_authors.user: talor-abr
paper_authors._id: 674d3de96ab71cc666cce981
paper_authors.name: Nir Ailon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:22:03.393Z
paper_authors._id: 66bf1cee43a701a83775810f
paper_authors.avatarUrl: /avatars/3b0cadcc77905facee1ae3511ec28999.svg
paper_authors.fullname: Nir Ailon
paper_authors.type: user
paper_authors.user: nailon-nvidia
paper_authors._id: 674d3de96ab71cc666cce982
paper_authors.name: Nave Assaf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:22:09.140Z
paper_authors._id: 64df9df14a8463e302eb69c6
paper_authors.avatarUrl: /avatars/db2151d46aaac236ce371cdd9d3b2302.svg
paper_authors.fullname: Nave Assaf
paper_authors.type: user
paper_authors.user: nav4
paper_authors._id: 674d3de96ab71cc666cce983
paper_authors.name: Mohammad Dabbah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:22:15.162Z
paper_authors._id: 63a4182f84a6a25c65bcfc71
paper_authors.avatarUrl: /avatars/9e9c700ed5b06a2388a9bb0e93d5beb1.svg
paper_authors.fullname: mohammad Dabbah
paper_authors.type: user
paper_authors.user: mdabbah
paper_authors._id: 674d3de96ab71cc666cce984
paper_authors.name: Ido Galil
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:22:30.830Z
paper_authors._id: 66867535e44c9931f329ca43
paper_authors.avatarUrl: /avatars/ec6fc05b9f596f1ce38ec061d3d1d230.svg
paper_authors.fullname: Ido Galil
paper_authors.type: user
paper_authors.user: IdoGalilNvidia
paper_authors._id: 674d3de96ab71cc666cce985
paper_authors.name: Amnon Geifman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:22:39.096Z
paper_authors._id: 65006cac12c1442d993d6d51
paper_authors.avatarUrl: /avatars/6700109303b902d453f3d8e2b45a103f.svg
paper_authors.fullname: Geifman
paper_authors.type: user
paper_authors.user: AmnonGeifman
paper_authors._id: 674d3de96ab71cc666cce986
paper_authors.name: Yonatan Geifman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:22:56.831Z
paper_authors._id: 66efe3f10989ae1ac1a1795f
paper_authors.avatarUrl: /avatars/c798905aeaa0fca51fb0f7b1edeabdc2.svg
paper_authors.fullname: Yonatan Geifman
paper_authors.type: user
paper_authors.user: ygeifman
paper_authors._id: 674d3de96ab71cc666cce987
paper_authors.name: Izhak Golan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-02T12:52:28.542Z
paper_authors._id: 66cb4b53d9772842192b84c3
paper_authors.avatarUrl: /avatars/f4820f60d509c3fa5f94ab72eff854c6.svg
paper_authors.fullname: Izik Golan
paper_authors.type: user
paper_authors.user: izikg
paper_authors._id: 674d3de96ab71cc666cce988
paper_authors.name: Netanel Haber
paper_authors._id: 674d3de96ab71cc666cce989
paper_authors.name: Ehud Karpas
paper_authors._id: 674d3de96ab71cc666cce98a
paper_authors.name: Itay Levy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:31:54.896Z
paper_authors._id: 668578fdd24e614fec97eac8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/668578fdd24e614fec97eac8/n5xYnqo5nQbVX2tgaRfEi.jpeg
paper_authors.fullname: Itay Levy
paper_authors.type: user
paper_authors.user: itlevy
paper_authors._id: 674d3de96ab71cc666cce98b
paper_authors.name: Shahar Mor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:32:05.610Z
paper_authors._id: 657018a04f3a65f869b10e21
paper_authors.avatarUrl: /avatars/715630dc868a728a7866d68fdf9f9d26.svg
paper_authors.fullname: Shahar Mor
paper_authors.type: user
paper_authors.user: ShaharM98
paper_authors._id: 674d3de96ab71cc666cce98c
paper_authors.name: Zach Moshe
paper_authors._id: 674d3de96ab71cc666cce98d
paper_authors.name: Najeeb Nabwani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:32:53.456Z
paper_authors._id: 6685860d852db86b9c91a1f3
paper_authors.avatarUrl: /avatars/ea25dd7427375b423af9c7b412826fef.svg
paper_authors.fullname: Najeeb Nabwani
paper_authors.type: user
paper_authors.user: Najeebnv
paper_authors._id: 674d3de96ab71cc666cce98e
paper_authors.name: Omri Puny
paper_authors._id: 674d3de96ab71cc666cce98f
paper_authors.name: Ran Rubin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:34:35.218Z
paper_authors._id: 65059361ffc738079c5d5583
paper_authors.avatarUrl: /avatars/03361adea86fa519941a79f8171d867f.svg
paper_authors.fullname: Ran Rubin
paper_authors.type: user
paper_authors.user: ranrubin
paper_authors._id: 674d3de96ab71cc666cce990
paper_authors.name: Itamar Schen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:34:29.156Z
paper_authors._id: 665f0a46f065b1d42806000d
paper_authors.avatarUrl: /avatars/927f042a3c95c5846621e2a381c66bbf.svg
paper_authors.fullname: Itamar Schen
paper_authors.type: user
paper_authors.user: ischen-nvidia
paper_authors._id: 674d3de96ab71cc666cce991
paper_authors.name: Ido Shahaf
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:34:21.829Z
paper_authors._id: 666ef13c14f1c262feeb706c
paper_authors.avatarUrl: /avatars/7dca59acf5e069d96bdbb98dace9199b.svg
paper_authors.fullname: Ido Shahaf
paper_authors.type: user
paper_authors.user: ishahaf
paper_authors._id: 674d3de96ab71cc666cce992
paper_authors.name: Oren Tropp
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:34:14.298Z
paper_authors._id: 66b089f14ae4ae811218cdb6
paper_authors.avatarUrl: /avatars/a50fe725922dfdbe0e731fade381b22e.svg
paper_authors.fullname: Oren Tropp
paper_authors.type: user
paper_authors.user: otropp
paper_authors._id: 674d3de96ab71cc666cce993
paper_authors.name: Omer Ullman Argov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:33:34.295Z
paper_authors._id: 66a0d0b2d4ab10840b5af2b0
paper_authors.avatarUrl: /avatars/8fa12818329de9457f93081e8f345117.svg
paper_authors.fullname: Omer Ullman Argov
paper_authors.type: user
paper_authors.user: oargov-nv
paper_authors._id: 674d3de96ab71cc666cce994
paper_authors.name: Ran Zilberstein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:33:26.752Z
paper_authors._id: 666027917c3f9c72113cc75c
paper_authors.avatarUrl: /avatars/a276ebe8e2731b6a05e3c61c2ae0ddae.svg
paper_authors.fullname: Ran Zilberstein
paper_authors.type: user
paper_authors.user: RanZilberstein-Nvidia
paper_authors._id: 674d3de96ab71cc666cce995
paper_authors.name: Ran El-Yaniv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T09:33:16.354Z
paper_authors._id: 66efd764155c986c6511b52d
paper_authors.avatarUrl: /avatars/711f9c6954a8623a9e014c0c563ca708.svg
paper_authors.fullname: Ran El-Yaniv
paper_authors.type: user
paper_authors.user: relyaniv
paper_title: Puzzle: Distillation-Based NAS for Inference-Optimized LLMs
paper_summary: Large language models (LLMs) have demonstrated remarkable capabilities, but
  their adoption is limited by high computational costs during inference. While
  increasing parameter counts enhances accuracy, it also widens the gap between
  state-of-the-art capabilities and practical deployability. We present Puzzle, a
  framework to accelerate LLM inference on specific hardware while preserving
  their capabilities. Through an innovative application of neural architecture
  search (NAS) at an unprecedented scale, Puzzle systematically optimizes models
  with tens of billions of parameters under hardware constraints. Our approach
  utilizes blockwise local knowledge distillation (BLD) for parallel architecture
  exploration and employs mixed-integer programming for precise constraint
  optimization.
    We demonstrate the real-world impact of our framework through
  Llama-3.1-Nemotron-51B-Instruct (Nemotron-51B), a publicly available model
  derived from Llama-3.1-70B-Instruct. Nemotron-51B achieves a 2.17x inference
  throughput speedup, fitting on a single NVIDIA H100 GPU while preserving 98.4%
  of the original model's capabilities. Nemotron-51B currently stands as the most
  accurate language model capable of inference on a single GPU with large batch
  sizes. Remarkably, this transformation required just 45B training tokens,
  compared to over 15T tokens used for the 70B model it was derived from. This
  establishes a new paradigm where powerful models can be optimized for efficient
  deployment with only negligible compromise of their capabilities, demonstrating
  that inference performance, not parameter count alone, should guide model
  selection. With the release of Nemotron-51B and the presentation of the Puzzle
  framework, we provide practitioners immediate access to state-of-the-art
  language modeling capabilities at significantly reduced computational costs.
paper_ai_summary: Puzzle is a framework that optimizes large language model inference using neural architecture search, blockwise local knowledge distillation, and mixed-integer programming to achieve faster throughput and reduced computational costs while preserving model accuracy.
paper_ai_keywords: neural architecture search blockwise local knowledge distillation mixed-integer programming large language models LLM inference throughput single GPU hardware constraints model capabilities state-of-the-art language model computational costs parameter count inference performance