LiveMind: Low-latency Large Language Models with Simultaneous Inference
In this paper, we introduce a novel low-latency inference framework for large
language models (LLMs) inference which enables LLMs to perform inferences with
incomplete prompts. By reallocating computational processes to prompt input
phase, we achieve a substantial reduction in latency, thereby significantly
enhancing the interactive experience for users of LLMs. The framework adeptly
manages the visibility of the streaming prompt to the model, allowing it to
infer from incomplete prompts or await additional prompts. Compared with
traditional inference methods that utilize complete prompts, our approach
demonstrates an average reduction of 59% in response latency on the MMLU-Pro
dataset, while maintaining comparable accuracy. Additionally, our framework
facilitates collaborative inference and output across different models. By
employing an LLM for inference and a small language model (SLM) for output, we
achieve an average 68% reduction in response latency, alongside a 5.5%
improvement in accuracy on the MMLU-Pro dataset compared with the SLM baseline.
For long prompts exceeding 20 sentences, the response latency can be reduced by
up to 93%.
6675457a9f2810b00961e366
Chuangtao Chen
extracted_confirmed
2024-06-26T10:50:50.784Z
667544fe25c64f9273795f30
/avatars/eb8abab4c8aee8f340dc55ebff60cc90.svg
Chuangtao Chen
user
ChuangtaoChen-TUM
6675457a9f2810b00961e367
Grace Li Zhang
6675457a9f2810b00961e368
Xunzhao Yin
6675457a9f2810b00961e369
Cheng Zhuo
6675457a9f2810b00961e36a
Ulf Schlichtmann
6675457a9f2810b00961e36b
Bing Li
LiveMind: Low-latency Large Language Models with Simultaneous Inference
In this paper, we introduce a novel low-latency inference framework for large
language models (LLMs) inference which enables LLMs to perform inferences with
incomplete prompts. By reallocating computational processes to prompt input
phase, we achieve a substantial reduction in latency, thereby significantly
enhancing the interactive experience for users of LLMs. The framework adeptly
manages the visibility of the streaming prompt to the model, allowing it to
infer from incomplete prompts or await additional prompts. Compared with
traditional inference methods that utilize complete prompts, our approach
demonstrates an average reduction of 59% in response latency on the MMLU-Pro
dataset, while maintaining comparable accuracy. Additionally, our framework
facilitates collaborative inference and output across different models. By
employing an LLM for inference and a small language model (SLM) for output, we
achieve an average 68% reduction in response latency, alongside a 5.5%
improvement in accuracy on the MMLU-Pro dataset compared with the SLM baseline.
For long prompts exceeding 20 sentences, the response latency can be reduced by
up to 93%.
A novel low-latency inference framework for large language models enables predictions with incomplete prompts, reducing response latency while maintaining or improving accuracy.
low-latency inference large language models (LLMs) incomplete prompts visibility management streaming prompt collaborative inference small language model (SLM) MMLU-Pro dataset