title: LiveMind: Low-latency Large Language Models with Simultaneous Inference
summary: In this paper, we introduce a novel low-latency inference framework for large
  language models (LLMs) inference which enables LLMs to perform inferences with
  incomplete prompts. By reallocating computational processes to prompt input
  phase, we achieve a substantial reduction in latency, thereby significantly
  enhancing the interactive experience for users of LLMs. The framework adeptly
  manages the visibility of the streaming prompt to the model, allowing it to
  infer from incomplete prompts or await additional prompts. Compared with
  traditional inference methods that utilize complete prompts, our approach
  demonstrates an average reduction of 59% in response latency on the MMLU-Pro
  dataset, while maintaining comparable accuracy. Additionally, our framework
  facilitates collaborative inference and output across different models. By
  employing an LLM for inference and a small language model (SLM) for output, we
  achieve an average 68% reduction in response latency, alongside a 5.5%
  improvement in accuracy on the MMLU-Pro dataset compared with the SLM baseline.
  For long prompts exceeding 20 sentences, the response latency can be reduced by
  up to 93%.
paper_authors._id: 6675457a9f2810b00961e366
paper_authors.name: Chuangtao Chen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-26T10:50:50.784Z
paper_authors._id: 667544fe25c64f9273795f30
paper_authors.avatarUrl: /avatars/eb8abab4c8aee8f340dc55ebff60cc90.svg
paper_authors.fullname: Chuangtao Chen
paper_authors.type: user
paper_authors.user: ChuangtaoChen-TUM
paper_authors._id: 6675457a9f2810b00961e367
paper_authors.name: Grace Li Zhang
paper_authors._id: 6675457a9f2810b00961e368
paper_authors.name: Xunzhao Yin
paper_authors._id: 6675457a9f2810b00961e369
paper_authors.name: Cheng Zhuo
paper_authors._id: 6675457a9f2810b00961e36a
paper_authors.name: Ulf Schlichtmann
paper_authors._id: 6675457a9f2810b00961e36b
paper_authors.name: Bing Li
paper_title: LiveMind: Low-latency Large Language Models with Simultaneous Inference
paper_summary: In this paper, we introduce a novel low-latency inference framework for large
  language models (LLMs) inference which enables LLMs to perform inferences with
  incomplete prompts. By reallocating computational processes to prompt input
  phase, we achieve a substantial reduction in latency, thereby significantly
  enhancing the interactive experience for users of LLMs. The framework adeptly
  manages the visibility of the streaming prompt to the model, allowing it to
  infer from incomplete prompts or await additional prompts. Compared with
  traditional inference methods that utilize complete prompts, our approach
  demonstrates an average reduction of 59% in response latency on the MMLU-Pro
  dataset, while maintaining comparable accuracy. Additionally, our framework
  facilitates collaborative inference and output across different models. By
  employing an LLM for inference and a small language model (SLM) for output, we
  achieve an average 68% reduction in response latency, alongside a 5.5%
  improvement in accuracy on the MMLU-Pro dataset compared with the SLM baseline.
  For long prompts exceeding 20 sentences, the response latency can be reduced by
  up to 93%.
paper_ai_summary: A novel low-latency inference framework for large language models enables predictions with incomplete prompts, reducing response latency while maintaining or improving accuracy.
paper_ai_keywords: low-latency inference large language models (LLMs) incomplete prompts visibility management streaming prompt collaborative inference small language model (SLM) MMLU-Pro dataset