Auditing Prompt Caching in Language Model APIs
Prompt caching in large language models (LLMs) results in data-dependent
timing variations: cached prompts are processed faster than non-cached prompts.
These timing differences introduce the risk of side-channel timing attacks. For
example, if the cache is shared across users, an attacker could identify cached
prompts from fast API response times to learn information about other users'
prompts. Because prompt caching may cause privacy leakage, transparency around
the caching policies of API providers is important. To this end, we develop and
conduct statistical audits to detect prompt caching in real-world LLM API
providers. We detect global cache sharing across users in seven API providers,
including OpenAI, resulting in potential privacy leakage about users' prompts.
Timing variations due to prompt caching can also result in leakage of
information about model architecture. Namely, we find evidence that OpenAI's
embedding model is a decoder-only Transformer, which was previously not
publicly known.
67ac1f7851c7f3b53ffc4def
Chenchen Gu
claimed_verified
2025-02-13T20:36:04.615Z
64b78d941d913123e6de2ecc
/avatars/f1225e5fc2033c6a473db8adcb911e3d.svg
Chenchen Gu
user
cygu
67ac1f7851c7f3b53ffc4df0
Xiang Lisa Li
67ac1f7851c7f3b53ffc4df1
Rohith Kuditipudi
67ac1f7851c7f3b53ffc4df2
Percy Liang
67ac1f7851c7f3b53ffc4df3
Tatsunori Hashimoto
extracted_pending
2025-02-12T04:11:36.912Z
661595d1b3d0b21da55cde7d
/avatars/ba3fa065536518637d21a5c46cee5dd1.svg
Tatsu Hashimoto
user
thashim
Auditing Prompt Caching in Language Model APIs
Prompt caching in large language models (LLMs) results in data-dependent
timing variations: cached prompts are processed faster than non-cached prompts.
These timing differences introduce the risk of side-channel timing attacks. For
example, if the cache is shared across users, an attacker could identify cached
prompts from fast API response times to learn information about other users'
prompts. Because prompt caching may cause privacy leakage, transparency around
the caching policies of API providers is important. To this end, we develop and
conduct statistical audits to detect prompt caching in real-world LLM API
providers. We detect global cache sharing across users in seven API providers,
including OpenAI, resulting in potential privacy leakage about users' prompts.
Timing variations due to prompt caching can also result in leakage of
information about model architecture. Namely, we find evidence that OpenAI's
embedding model is a decoder-only Transformer, which was previously not
publicly known.
Statistical audits detect prompt caching across users in LLM APIs, revealing privacy risks and model architecture details.
prompt caching LLM API providers statistical audits side-channel timing attacks Transformer decoder-only model