Distributed Speculative Inference of Large Language Models
Accelerating the inference of large language models (LLMs) is an important
challenge in artificial intelligence. This paper introduces distributed
speculative inference (DSI), a novel distributed inference algorithm that is
provably faster than speculative inference (SI) [leviathan2023fast,
chen2023accelerating, miao2023specinfer] and traditional autoregressive
inference (non-SI). Like other SI algorithms, DSI works on frozen LLMs,
requiring no training or architectural modifications, and it preserves the
target distribution.
  Prior studies on SI have demonstrated empirical speedups (compared to non-SI)
but require a fast and accurate drafter LLM. In practice, off-the-shelf LLMs
often do not have matching drafters that are sufficiently fast and accurate. We
show a gap: SI gets slower than non-SI when using slower or less accurate
drafters. We close this gap by proving that DSI is faster than both SI and
non-SI given any drafters. By orchestrating multiple instances of the target
and drafters, DSI is not only faster than SI but also supports LLMs that cannot
be accelerated with SI.
  Our simulations show speedups of off-the-shelf LLMs in realistic settings:
DSI is 1.29-1.92x faster than SI.
66501f22e58928eeec0781f6
Nadav Timor
admin_assigned
2024-05-24T11:59:53.913Z
62be5fc0c990a0a515da5f34
https://cdn-avatars.huggingface.co/v1/production/uploads/1674221847840-62be5fc0c990a0a515da5f34.jpeg
Nadav Timor
user
Nadav-Timor
66501f22e58928eeec0781f7
Jonathan Mamou
admin_assigned
2024-05-24T12:00:01.742Z
606d6349f1259f30578520ad
https://cdn-avatars.huggingface.co/v1/production/uploads/606d6349f1259f30578520ad/72_XrFfgQ6p9tgJj5Bc5U.png
Jonathan Mamou
user
jmamou
66501f22e58928eeec0781f8
Daniel Korat
admin_assigned
2024-05-24T12:00:08.265Z
6055ae5d25cd24537dd59dc5
https://cdn-avatars.huggingface.co/v1/production/uploads/6055ae5d25cd24537dd59dc5/eswozkCirLrnyhufN8_-f.jpeg
Daniel Korat
user
danielkorat
66501f22e58928eeec0781f9
Moshe Berchansky
admin_assigned
2024-05-24T12:00:33.796Z
63e0c8875c6964861ebb0c49
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0c8875c6964861ebb0c49/yzkhPSxgXtJCM62iMBOOK.jpeg
Moshe Berchansky
user
mber
66501f22e58928eeec0781fa
Oren Pereg
admin_assigned
2024-05-24T12:00:40.601Z
60570320cbe9c7542f3501e3
https://cdn-avatars.huggingface.co/v1/production/uploads/1664643955283-60570320cbe9c7542f3501e3.jpeg
Oren Pereg
user
orenpereg
66501f22e58928eeec0781fb
Moshe Wasserblat
admin_assigned
2024-05-24T12:00:46.779Z
6013c64031d84a3bb7650038
/avatars/e05ca715004b39e79472399f75010bda.svg
Moshe Wasserblat
user
moshew
66501f22e58928eeec0781fc
Tomer Galanti
admin_assigned
2024-05-24T12:00:53.077Z
64efb071c23e876cacba9cfe
/avatars/0c3c78ba3daf41177d78294727292c56.svg
Tomer Galanti
user
TomerGalanti
66501f22e58928eeec0781fd
Michal Gordon
66501f22e58928eeec0781fe
David Harel
Distributed Speculative Inference of Large Language Models
Accelerating the inference of large language models (LLMs) is an important
challenge in artificial intelligence. This paper introduces distributed
speculative inference (DSI), a novel distributed inference algorithm that is
provably faster than speculative inference (SI) [leviathan2023fast,
chen2023accelerating, miao2023specinfer] and traditional autoregressive
inference (non-SI). Like other SI algorithms, DSI works on frozen LLMs,
requiring no training or architectural modifications, and it preserves the
target distribution.
  Prior studies on SI have demonstrated empirical speedups (compared to non-SI)
but require a fast and accurate drafter LLM. In practice, off-the-shelf LLMs
often do not have matching drafters that are sufficiently fast and accurate. We
show a gap: SI gets slower than non-SI when using slower or less accurate
drafters. We close this gap by proving that DSI is faster than both SI and
non-SI given any drafters. By orchestrating multiple instances of the target
and drafters, DSI is not only faster than SI but also supports LLMs that cannot
be accelerated with SI.
  Our simulations show speedups of off-the-shelf LLMs in realistic settings:
DSI is 1.29-1.92x faster than SI.
Distributed speculative inference (DSI) accelerates large language model inference faster than speculative inference (SI) and traditional inference methods, supporting a wider range of drafters and achieving significant speedups.
distributed speculative inference speculative inference large language models autoregressive inference drafter LLMs