title: XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
summary: In-context learning (ICL) approaches typically leverage prompting to
  condition decoder-only language model generation on reference information.
  Just-in-time processing of a context is inefficient due to the quadratic cost
  of self-attention operations, and caching is desirable. However, caching
  transformer states can easily require almost as much space as the model
  parameters. When the right context isn't known in advance, caching ICL can be
  challenging. This work addresses these limitations by introducing models that,
  inspired by the encoder-decoder architecture, use cross-attention to condition
  generation on reference text without the prompt. More precisely, we leverage
  pre-trained decoder-only models and only train a small number of added layers.
  We use Question-Answering (QA) as a testbed to evaluate the ability of our
  models to perform conditional generation and observe that they outperform ICL,
  are comparable to fine-tuned prompted LLMs, and drastically reduce the space
  footprint relative to standard KV caching by two orders of magnitude.
paper_authors._id: 662a97804eaa187e4cf2a7fc
paper_authors.name: João Monteiro
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:30:18.867Z
paper_authors._id: 63347fed495073b88704cf02
paper_authors.avatarUrl: /avatars/cb5545adaf1cfe0166119c586f4ac9c8.svg
paper_authors.fullname: João Monteiro
paper_authors.type: user
paper_authors.user: joaomonteirof
paper_authors._id: 662a97804eaa187e4cf2a7fd
paper_authors.name: Étienne Marcotte
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T13:44:36.539Z
paper_authors._id: 64ac5d977b5ff76277f146bd
paper_authors.avatarUrl: /avatars/d01ca3ecc554c6a8e4d014010d111e6f.svg
paper_authors.fullname: Étienne Marcotte
paper_authors.type: user
paper_authors.user: marcotet
paper_authors._id: 662a97804eaa187e4cf2a7fe
paper_authors.name: Pierre-André Noël
paper_authors._id: 662a97804eaa187e4cf2a7ff
paper_authors.name: Valentina Zantedeschi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T13:44:57.573Z
paper_authors._id: 6454b7601a543cf97b1add68
paper_authors.avatarUrl: /avatars/485ca04a5353046ebf3972357a17f976.svg
paper_authors.fullname: Valentina Zantedeschi
paper_authors.type: user
paper_authors.user: vzantedeschi
paper_authors._id: 662a97804eaa187e4cf2a800
paper_authors.name: David Vázquez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T13:45:19.875Z
paper_authors._id: 648982952fdd12195be408ad
paper_authors.avatarUrl: /avatars/33b7c11f3250ebbfa426b6f63208af25.svg
paper_authors.fullname: David Vázquez
paper_authors.type: user
paper_authors.user: microbode
paper_authors._id: 662a97804eaa187e4cf2a801
paper_authors.name: Nicolas Chapados
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T13:45:26.313Z
paper_authors._id: 631f54aa5ba8c026340b13cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
paper_authors.fullname: Nicolas Chapados
paper_authors.type: user
paper_authors.user: nicolaschapados
paper_authors._id: 662a97804eaa187e4cf2a802
paper_authors.name: Christopher Pal
paper_authors._id: 662a97804eaa187e4cf2a803
paper_authors.name: Perouz Taslakian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T15:28:44.706Z
paper_authors._id: 648b43f8085e3bb616eed139
paper_authors.avatarUrl: /avatars/41487af3a14507fb6c21885e9e57172c.svg
paper_authors.fullname: Perouz Taslakian
paper_authors.type: user
paper_authors.user: Perouz
paper_title: XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
paper_summary: In-context learning (ICL) approaches typically leverage prompting to
  condition decoder-only language model generation on reference information.
  Just-in-time processing of a context is inefficient due to the quadratic cost
  of self-attention operations, and caching is desirable. However, caching
  transformer states can easily require almost as much space as the model
  parameters. When the right context isn't known in advance, caching ICL can be
  challenging. This work addresses these limitations by introducing models that,
  inspired by the encoder-decoder architecture, use cross-attention to condition
  generation on reference text without the prompt. More precisely, we leverage
  pre-trained decoder-only models and only train a small number of added layers.
  We use Question-Answering (QA) as a testbed to evaluate the ability of our
  models to perform conditional generation and observe that they outperform ICL,
  are comparable to fine-tuned prompted LLMs, and drastically reduce the space
  footprint relative to standard KV caching by two orders of magnitude.
paper_ai_summary: Models using cross-attention for context conditioning in language generation outperform in-context learning and reduce memory usage compared to traditional caching methods.
paper_ai_keywords: in-context learning (ICL) prompting decoder-only language model self-attention transformer states cross-attention encoder-decoder architecture pre-trained decoder-only models Question-Answering (QA) fine-tuned prompted LLMs KV caching