XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
In-context learning (ICL) approaches typically leverage prompting to
condition decoder-only language model generation on reference information.
Just-in-time processing of a context is inefficient due to the quadratic cost
of self-attention operations, and caching is desirable. However, caching
transformer states can easily require almost as much space as the model
parameters. When the right context isn't known in advance, caching ICL can be
challenging. This work addresses these limitations by introducing models that,
inspired by the encoder-decoder architecture, use cross-attention to condition
generation on reference text without the prompt. More precisely, we leverage
pre-trained decoder-only models and only train a small number of added layers.
We use Question-Answering (QA) as a testbed to evaluate the ability of our
models to perform conditional generation and observe that they outperform ICL,
are comparable to fine-tuned prompted LLMs, and drastically reduce the space
footprint relative to standard KV caching by two orders of magnitude.
662a97804eaa187e4cf2a7fc
João Monteiro
claimed_verified
2024-04-28T19:30:18.867Z
63347fed495073b88704cf02
/avatars/cb5545adaf1cfe0166119c586f4ac9c8.svg
João Monteiro
user
joaomonteirof
662a97804eaa187e4cf2a7fd
Étienne Marcotte
admin_assigned
2024-04-26T13:44:36.539Z
64ac5d977b5ff76277f146bd
/avatars/d01ca3ecc554c6a8e4d014010d111e6f.svg
Étienne Marcotte
user
marcotet
662a97804eaa187e4cf2a7fe
Pierre-André Noël
662a97804eaa187e4cf2a7ff
Valentina Zantedeschi
admin_assigned
2024-04-26T13:44:57.573Z
6454b7601a543cf97b1add68
/avatars/485ca04a5353046ebf3972357a17f976.svg
Valentina Zantedeschi
user
vzantedeschi
662a97804eaa187e4cf2a800
David Vázquez
admin_assigned
2024-04-26T13:45:19.875Z
648982952fdd12195be408ad
/avatars/33b7c11f3250ebbfa426b6f63208af25.svg
David Vázquez
user
microbode
662a97804eaa187e4cf2a801
Nicolas Chapados
admin_assigned
2024-04-26T13:45:26.313Z
631f54aa5ba8c026340b13cf
https://cdn-avatars.huggingface.co/v1/production/uploads/631f54aa5ba8c026340b13cf/2jI0VUDG5cKkdf2C5KJuy.png
Nicolas Chapados
user
nicolaschapados
662a97804eaa187e4cf2a802
Christopher Pal
662a97804eaa187e4cf2a803
Perouz Taslakian
claimed_verified
2024-04-26T15:28:44.706Z
648b43f8085e3bb616eed139
/avatars/41487af3a14507fb6c21885e9e57172c.svg
Perouz Taslakian
user
Perouz
XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
In-context learning (ICL) approaches typically leverage prompting to
condition decoder-only language model generation on reference information.
Just-in-time processing of a context is inefficient due to the quadratic cost
of self-attention operations, and caching is desirable. However, caching
transformer states can easily require almost as much space as the model
parameters. When the right context isn't known in advance, caching ICL can be
challenging. This work addresses these limitations by introducing models that,
inspired by the encoder-decoder architecture, use cross-attention to condition
generation on reference text without the prompt. More precisely, we leverage
pre-trained decoder-only models and only train a small number of added layers.
We use Question-Answering (QA) as a testbed to evaluate the ability of our
models to perform conditional generation and observe that they outperform ICL,
are comparable to fine-tuned prompted LLMs, and drastically reduce the space
footprint relative to standard KV caching by two orders of magnitude.
Models using cross-attention for context conditioning in language generation outperform in-context learning and reduce memory usage compared to traditional caching methods.
in-context learning (ICL) prompting decoder-only language model self-attention transformer states cross-attention encoder-decoder architecture pre-trained decoder-only models Question-Answering (QA) fine-tuned prompted LLMs KV caching