title: Auditing Prompt Caching in Language Model APIs
summary: Prompt caching in large language models (LLMs) results in data-dependent
  timing variations: cached prompts are processed faster than non-cached prompts.
  These timing differences introduce the risk of side-channel timing attacks. For
  example, if the cache is shared across users, an attacker could identify cached
  prompts from fast API response times to learn information about other users'
  prompts. Because prompt caching may cause privacy leakage, transparency around
  the caching policies of API providers is important. To this end, we develop and
  conduct statistical audits to detect prompt caching in real-world LLM API
  providers. We detect global cache sharing across users in seven API providers,
  including OpenAI, resulting in potential privacy leakage about users' prompts.
  Timing variations due to prompt caching can also result in leakage of
  information about model architecture. Namely, we find evidence that OpenAI's
  embedding model is a decoder-only Transformer, which was previously not
  publicly known.
paper_authors._id: 67ac1f7851c7f3b53ffc4def
paper_authors.name: Chenchen Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-13T20:36:04.615Z
paper_authors._id: 64b78d941d913123e6de2ecc
paper_authors.avatarUrl: /avatars/f1225e5fc2033c6a473db8adcb911e3d.svg
paper_authors.fullname: Chenchen Gu
paper_authors.type: user
paper_authors.user: cygu
paper_authors._id: 67ac1f7851c7f3b53ffc4df0
paper_authors.name: Xiang Lisa Li
paper_authors._id: 67ac1f7851c7f3b53ffc4df1
paper_authors.name: Rohith Kuditipudi
paper_authors._id: 67ac1f7851c7f3b53ffc4df2
paper_authors.name: Percy Liang
paper_authors._id: 67ac1f7851c7f3b53ffc4df3
paper_authors.name: Tatsunori Hashimoto
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-12T04:11:36.912Z
paper_authors._id: 661595d1b3d0b21da55cde7d
paper_authors.avatarUrl: /avatars/ba3fa065536518637d21a5c46cee5dd1.svg
paper_authors.fullname: Tatsu Hashimoto
paper_authors.type: user
paper_authors.user: thashim
paper_title: Auditing Prompt Caching in Language Model APIs
paper_summary: Prompt caching in large language models (LLMs) results in data-dependent
  timing variations: cached prompts are processed faster than non-cached prompts.
  These timing differences introduce the risk of side-channel timing attacks. For
  example, if the cache is shared across users, an attacker could identify cached
  prompts from fast API response times to learn information about other users'
  prompts. Because prompt caching may cause privacy leakage, transparency around
  the caching policies of API providers is important. To this end, we develop and
  conduct statistical audits to detect prompt caching in real-world LLM API
  providers. We detect global cache sharing across users in seven API providers,
  including OpenAI, resulting in potential privacy leakage about users' prompts.
  Timing variations due to prompt caching can also result in leakage of
  information about model architecture. Namely, we find evidence that OpenAI's
  embedding model is a decoder-only Transformer, which was previously not
  publicly known.
paper_ai_keywords: **LLMs (Large Language Models)** **prompt caching** **side-channel timing attacks** **API providers** **Transparency** **perform audits** **embedding model** **decoder-only Transformer**