Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
Large language models (LLMs) often hallucinate and lack the ability to
provide attribution for their generations. Semi-parametric LMs, such as kNN-LM,
approach these limitations by refining the output of an LM for a given prompt
using its nearest neighbor matches in a non-parametric data store. However,
these models often exhibit slow inference speeds and produce non-fluent texts.
In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a
novel semi-parametric language modeling approach that is capable of
incorporating real-world text spans of arbitrary length into the LM generations
and providing attribution to their sources. NEST performs token-level retrieval
at each inference step to compute a semi-parametric mixture distribution and
identify promising span continuations in a corpus. It then uses an approximate
speculative decoding procedure that accepts a prefix of the retrieved span or
generates a new token. NEST significantly enhances the generation quality and
attribution rate of the base LM across a variety of knowledge-intensive tasks,
surpassing the conventional kNN-LM method and performing competitively with
in-context retrieval augmentation. In addition, NEST substantially improves the
generation speed, achieving a 1.8x speedup in inference time when applied to
Llama-2-Chat 70B.
665887f61c264957e5a76381
Minghan Li
admin_assigned
2024-05-31T10:52:17.203Z
61340b5e2d28e0a2db921ee3
/avatars/5c1be8985ef2e68ca2508f9c1a9ff1f9.svg
Minghan Li
user
alexlimh
665887f61c264957e5a76382
Xilun Chen
admin_assigned
2024-05-31T10:52:34.754Z
62ffccd18813dd8628172c6c
/avatars/62741be5967a74758a2b7914aa842ec8.svg
Xilun Chen
user
ccsasuke
665887f61c264957e5a76383
Ari Holtzman
admin_assigned
2024-05-31T10:53:01.346Z
6508c81db44445e9b36f3a1e
/avatars/7d4b257398c601b482b9655f046021f3.svg
Ari Holtzman
user
ariholtzman
665887f61c264957e5a76384
Beidi Chen
admin_assigned
2024-05-31T10:53:07.097Z
64b732f832403871593e082c
/avatars/dd21932b0c167131ee7545a622c46c3c.svg
Beidi Chen
user
beidic
665887f61c264957e5a76385
Jimmy Lin
665887f61c264957e5a76386
Wen-tau Yih
665887f61c264957e5a76387
Xi Victoria Lin
admin_assigned
2024-05-31T10:54:51.458Z
61df349e34b589fbaf2e93c4
/avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
Victoria Lin
user
VictoriaLinML
Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
Large language models (LLMs) often hallucinate and lack the ability to
provide attribution for their generations. Semi-parametric LMs, such as kNN-LM,
approach these limitations by refining the output of an LM for a given prompt
using its nearest neighbor matches in a non-parametric data store. However,
these models often exhibit slow inference speeds and produce non-fluent texts.
In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a
novel semi-parametric language modeling approach that is capable of
incorporating real-world text spans of arbitrary length into the LM generations
and providing attribution to their sources. NEST performs token-level retrieval
at each inference step to compute a semi-parametric mixture distribution and
identify promising span continuations in a corpus. It then uses an approximate
speculative decoding procedure that accepts a prefix of the retrieved span or
generates a new token. NEST significantly enhances the generation quality and
attribution rate of the base LM across a variety of knowledge-intensive tasks,
surpassing the conventional kNN-LM method and performing competitively with
in-context retrieval augmentation. In addition, NEST substantially improves the
generation speed, achieving a 1.8x speedup in inference time when applied to
Llama-2-Chat 70B.
Nearest Neighbor Speculative Decoding enhances language model generation quality, attribution, and speed through token-level retrieval and speculative decoding.
semi-parametric LMs kNN-LM Nearest Neighbor Speculative Decoding token-level retrieval semi-parametric mixture distribution speculative decoding generation quality attribution rate knowledge-intensive tasks in-context retrieval augmentation inference speed