title: Adaptive Semantic Prompt Caching with VectorQ
summary: Semantic prompt caches reduce the latency and cost of large language model
  (LLM) inference by reusing cached LLM-generated responses for semantically
  similar prompts. Vector similarity metrics assign a numerical score to quantify
  the similarity between an embedded prompt and its nearest neighbor in the
  cache. Existing systems rely on a static threshold to classify whether the
  similarity score is sufficiently high to result in a cache hit. We show that
  this one-size-fits-all threshold is insufficient across different prompts. We
  propose VectorQ, a framework to learn embedding-specific threshold regions that
  adapt to the complexity and uncertainty of an embedding. Through evaluations on
  a combination of four diverse datasets, we show that VectorQ consistently
  outperforms state-of-the-art systems across all static thresholds, achieving up
  to 12x increases in cache hit rate and error rate reductions up to 92%.
paper_authors._id: 67aaa0ebe37429ebdbd113cf
paper_authors.name: Luis Gaspar Schroeder
paper_authors._id: 67aaa0ebe37429ebdbd113d0
paper_authors.name: Shu Liu
paper_authors._id: 67aaa0ebe37429ebdbd113d1
paper_authors.name: Alejandro Cuadron
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:36.306Z
paper_authors._id: 652a656d1a3250bbfe3bb92d
paper_authors.avatarUrl: /avatars/a1c25150d55c493edd9a7f81287fc449.svg
paper_authors.fullname: Alejandro Cuadron Lafuente
paper_authors.type: user
paper_authors.user: AlexCuadron
paper_authors._id: 67aaa0ebe37429ebdbd113d2
paper_authors.name: Mark Zhao
paper_authors._id: 67aaa0ebe37429ebdbd113d3
paper_authors.name: Stephan Krusche
paper_authors._id: 67aaa0ebe37429ebdbd113d4
paper_authors.name: Alfons Kemper
paper_authors._id: 67aaa0ebe37429ebdbd113d5
paper_authors.name: Matei Zaharia
paper_authors._id: 67aaa0ebe37429ebdbd113d6
paper_authors.name: Joseph E. Gonzalez
paper_title: Adaptive Semantic Prompt Caching with VectorQ
paper_summary: Semantic prompt caches reduce the latency and cost of large language model
  (LLM) inference by reusing cached LLM-generated responses for semantically
  similar prompts. Vector similarity metrics assign a numerical score to quantify
  the similarity between an embedded prompt and its nearest neighbor in the
  cache. Existing systems rely on a static threshold to classify whether the
  similarity score is sufficiently high to result in a cache hit. We show that
  this one-size-fits-all threshold is insufficient across different prompts. We
  propose VectorQ, a framework to learn embedding-specific threshold regions that
  adapt to the complexity and uncertainty of an embedding. Through evaluations on
  a combination of four diverse datasets, we show that VectorQ consistently
  outperforms state-of-the-art systems across all static thresholds, achieving up
  to 12x increases in cache hit rate and error rate reductions up to 92%.
paper_ai_summary: VectorQ learns adaptive threshold regions for semantic prompt caching, significantly improving cache hit rates and error reductions in large language model inference.
paper_ai_keywords: semantic prompt caches latency cost large language models LLM inference vector similarity metrics cache hit embedding-specific threshold regions complexity uncertainty embedding cache hit rate error rate