GLIMMER: generalized late-interaction memory reranker
Memory-augmentation is a powerful approach for efficiently incorporating
external information into language models, but leads to reduced performance
relative to retrieving text. Recent work introduced LUMEN, a memory-retrieval
hybrid that partially pre-computes memory and updates memory representations on
the fly with a smaller live encoder.
  We propose GLIMMER, which improves on this approach through 1) exploiting
free access to the powerful memory representations by applying a shallow
reranker on top of memory to drastically improve retrieval quality at low cost,
and 2) incorporating multi-task training to learn a general and higher quality
memory and live encoder. GLIMMER achieves strong gains in performance at faster
speeds compared to LUMEN and FiD on the KILT benchmark of knowledge-intensive
tasks.
64925bd6a63ac82b347cb34e
Michiel de Jong
64925bd6a63ac82b347cb34f
Yury Zemlyanskiy
64925bd6a63ac82b347cb350
Nicholas FitzGerald
64925bd6a63ac82b347cb351
Sumit Sanghai
64925bd6a63ac82b347cb352
William W. Cohen
64925bd6a63ac82b347cb353
Joshua Ainslie
GLIMMER: generalized late-interaction memory reranker
Memory-augmentation is a powerful approach for efficiently incorporating
external information into language models, but leads to reduced performance
relative to retrieving text. Recent work introduced LUMEN, a memory-retrieval
hybrid that partially pre-computes memory and updates memory representations on
the fly with a smaller live encoder.
  We propose GLIMMER, which improves on this approach through 1) exploiting
free access to the powerful memory representations by applying a shallow
reranker on top of memory to drastically improve retrieval quality at low cost,
and 2) incorporating multi-task training to learn a general and higher quality
memory and live encoder. GLIMMER achieves strong gains in performance at faster
speeds compared to LUMEN and FiD on the KILT benchmark of knowledge-intensive
tasks.
GLIMMER enhances memory-augmented language models by using a shallow reranker and multi-task training, improving retrieval quality and speed over LUMEN and FiD on knowledge-intensive tasks.
memory-augmentation language models LUMEN memory-retrieval pre-computes memory representations live encoder shallow reranker multi-task training KILT benchmark