title: Prompt Cache: Modular Attention Reuse for Low-Latency Inference
summary: We present Prompt Cache, an approach for accelerating inference for large
  language models (LLM) by reusing attention states across different LLM prompts.
  Many input prompts have overlapping text segments, such as system messages,
  prompt templates, and documents provided for context. Our key insight is that
  by precomputing and storing the attention states of these frequently occurring
  text segments on the inference server, we can efficiently reuse them when these
  segments appear in user prompts. Prompt Cache employs a schema to explicitly
  define such reusable text segments, called prompt modules. The schema ensures
  positional accuracy during attention state reuse and provides users with an
  interface to access cached states in their prompt. Using a prototype
  implementation, we evaluate Prompt Cache across several LLMs. We show that
  Prompt Cache significantly reduce latency in time-to-first-token, especially
  for longer prompts such as document-based question answering and
  recommendations. The improvements range from 8x for GPU-based inference to 60x
  for CPU-based inference, all while maintaining output accuracy and without the
  need for model parameter modifications.
paper_authors._id: 654dbb1bca2f3d20259209be
paper_authors.name: In Gim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:39:55.008Z
paper_authors._id: 63d31d672727d7888cbc2428
paper_authors.avatarUrl: /avatars/1df66789dd7ed8f3c068f9d440860ed9.svg
paper_authors.fullname: In Gim
paper_authors.type: user
paper_authors.user: ingim
paper_authors._id: 654dbb1bca2f3d20259209bf
paper_authors.name: Guojun Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-10T18:39:41.660Z
paper_authors._id: 6500693c051fae19fc4c2c1b
paper_authors.avatarUrl: /avatars/10fe24e1eabde8016b7c95a53ac153b7.svg
paper_authors.fullname: Guojun Chen
paper_authors.type: user
paper_authors.user: Leonana69
paper_authors._id: 654dbb1bca2f3d20259209c0
paper_authors.name: Seung-seob Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:40:25.582Z
paper_authors._id: 64bec1fd9f94ea2554258be0
paper_authors.avatarUrl: /avatars/13d8a12f937ab5a95c2c1bfdb136aee9.svg
paper_authors.fullname: Seung-seob Lee
paper_authors.type: user
paper_authors.user: seungseob7lee
paper_authors._id: 654dbb1bca2f3d20259209c1
paper_authors.name: Nikhil Sarda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:40:32.989Z
paper_authors._id: 64f9a619a5067f6b65542cbf
paper_authors.avatarUrl: /avatars/d0ebeeb7123a8d5f432970e2efb6cd6a.svg
paper_authors.fullname: Nikhil Sarda
paper_authors.type: user
paper_authors.user: diffoperator
paper_authors._id: 654dbb1bca2f3d20259209c2
paper_authors.name: Anurag Khandelwal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-10T10:40:39.747Z
paper_authors._id: 644d68cf5c1b4e14d0c20be9
paper_authors.avatarUrl: /avatars/0dd280cbb75951834a013521734124ec.svg
paper_authors.fullname: Anurag Khandelwal 
paper_authors.type: user
paper_authors.user: Anurag88
paper_authors._id: 654dbb1bca2f3d20259209c3
paper_authors.name: Lin Zhong
paper_title: Prompt Cache: Modular Attention Reuse for Low-Latency Inference
paper_summary: We present Prompt Cache, an approach for accelerating inference for large
  language models (LLM) by reusing attention states across different LLM prompts.
  Many input prompts have overlapping text segments, such as system messages,
  prompt templates, and documents provided for context. Our key insight is that
  by precomputing and storing the attention states of these frequently occurring
  text segments on the inference server, we can efficiently reuse them when these
  segments appear in user prompts. Prompt Cache employs a schema to explicitly
  define such reusable text segments, called prompt modules. The schema ensures
  positional accuracy during attention state reuse and provides users with an
  interface to access cached states in their prompt. Using a prototype
  implementation, we evaluate Prompt Cache across several LLMs. We show that
  Prompt Cache significantly reduce latency in time-to-first-token, especially
  for longer prompts such as document-based question answering and
  recommendations. The improvements range from 8x for GPU-based inference to 60x
  for CPU-based inference, all while maintaining output accuracy and without the
  need for model parameter modifications.
paper_ai_keywords: attention states large language models (LLM) system messages prompt templates documents prompt modules positional accuracy time-to-first-token document-based question answering recommendations