title: Distributed Speculative Inference of Large Language Models
summary: Accelerating the inference of large language models (LLMs) is an important
  challenge in artificial intelligence. This paper introduces distributed
  speculative inference (DSI), a novel distributed inference algorithm that is
  provably faster than speculative inference (SI) [leviathan2023fast,
  chen2023accelerating, miao2023specinfer] and traditional autoregressive
  inference (non-SI). Like other SI algorithms, DSI works on frozen LLMs,
  requiring no training or architectural modifications, and it preserves the
  target distribution.
    Prior studies on SI have demonstrated empirical speedups (compared to non-SI)
  but require a fast and accurate drafter LLM. In practice, off-the-shelf LLMs
  often do not have matching drafters that are sufficiently fast and accurate. We
  show a gap: SI gets slower than non-SI when using slower or less accurate
  drafters. We close this gap by proving that DSI is faster than both SI and
  non-SI given any drafters. By orchestrating multiple instances of the target
  and drafters, DSI is not only faster than SI but also supports LLMs that cannot
  be accelerated with SI.
    Our simulations show speedups of off-the-shelf LLMs in realistic settings:
  DSI is 1.29-1.92x faster than SI.
paper_authors._id: 66501f22e58928eeec0781f6
paper_authors.name: Nadav Timor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:59:53.913Z
paper_authors._id: 62be5fc0c990a0a515da5f34
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674221847840-62be5fc0c990a0a515da5f34.jpeg
paper_authors.fullname: Nadav Timor
paper_authors.type: user
paper_authors.user: Nadav-Timor
paper_authors._id: 66501f22e58928eeec0781f7
paper_authors.name: Jonathan Mamou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:00:01.742Z
paper_authors._id: 606d6349f1259f30578520ad
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/606d6349f1259f30578520ad/72_XrFfgQ6p9tgJj5Bc5U.png
paper_authors.fullname: Jonathan Mamou
paper_authors.type: user
paper_authors.user: jmamou
paper_authors._id: 66501f22e58928eeec0781f8
paper_authors.name: Daniel Korat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:00:08.265Z
paper_authors._id: 6055ae5d25cd24537dd59dc5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6055ae5d25cd24537dd59dc5/eswozkCirLrnyhufN8_-f.jpeg
paper_authors.fullname: Daniel Korat
paper_authors.type: user
paper_authors.user: danielkorat
paper_authors._id: 66501f22e58928eeec0781f9
paper_authors.name: Moshe Berchansky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:00:33.796Z
paper_authors._id: 63e0c8875c6964861ebb0c49
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e0c8875c6964861ebb0c49/yzkhPSxgXtJCM62iMBOOK.jpeg
paper_authors.fullname: Moshe Berchansky
paper_authors.type: user
paper_authors.user: mber
paper_authors._id: 66501f22e58928eeec0781fa
paper_authors.name: Oren Pereg
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:00:40.601Z
paper_authors._id: 60570320cbe9c7542f3501e3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664643955283-60570320cbe9c7542f3501e3.jpeg
paper_authors.fullname: Oren Pereg
paper_authors.type: user
paper_authors.user: orenpereg
paper_authors._id: 66501f22e58928eeec0781fb
paper_authors.name: Moshe Wasserblat
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:00:46.779Z
paper_authors._id: 6013c64031d84a3bb7650038
paper_authors.avatarUrl: /avatars/e05ca715004b39e79472399f75010bda.svg
paper_authors.fullname: Moshe Wasserblat
paper_authors.type: user
paper_authors.user: moshew
paper_authors._id: 66501f22e58928eeec0781fc
paper_authors.name: Tomer Galanti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T12:00:53.077Z
paper_authors._id: 64efb071c23e876cacba9cfe
paper_authors.avatarUrl: /avatars/0c3c78ba3daf41177d78294727292c56.svg
paper_authors.fullname: Tomer Galanti
paper_authors.type: user
paper_authors.user: TomerGalanti
paper_authors._id: 66501f22e58928eeec0781fd
paper_authors.name: Michal Gordon
paper_authors._id: 66501f22e58928eeec0781fe
paper_authors.name: David Harel
paper_title: Distributed Speculative Inference of Large Language Models
paper_summary: Accelerating the inference of large language models (LLMs) is an important
  challenge in artificial intelligence. This paper introduces distributed
  speculative inference (DSI), a novel distributed inference algorithm that is
  provably faster than speculative inference (SI) [leviathan2023fast,
  chen2023accelerating, miao2023specinfer] and traditional autoregressive
  inference (non-SI). Like other SI algorithms, DSI works on frozen LLMs,
  requiring no training or architectural modifications, and it preserves the
  target distribution.
    Prior studies on SI have demonstrated empirical speedups (compared to non-SI)
  but require a fast and accurate drafter LLM. In practice, off-the-shelf LLMs
  often do not have matching drafters that are sufficiently fast and accurate. We
  show a gap: SI gets slower than non-SI when using slower or less accurate
  drafters. We close this gap by proving that DSI is faster than both SI and
  non-SI given any drafters. By orchestrating multiple instances of the target
  and drafters, DSI is not only faster than SI but also supports LLMs that cannot
  be accelerated with SI.
    Our simulations show speedups of off-the-shelf LLMs in realistic settings:
  DSI is 1.29-1.92x faster than SI.
paper_ai_keywords: distributed speculative inference (DSI) speculative inference (SI) autoregressive inference frozen LLMs drafter LLM target distribution