title: Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
summary: Large language models (LLMs) often hallucinate and lack the ability to
  provide attribution for their generations. Semi-parametric LMs, such as kNN-LM,
  approach these limitations by refining the output of an LM for a given prompt
  using its nearest neighbor matches in a non-parametric data store. However,
  these models often exhibit slow inference speeds and produce non-fluent texts.
  In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a
  novel semi-parametric language modeling approach that is capable of
  incorporating real-world text spans of arbitrary length into the LM generations
  and providing attribution to their sources. NEST performs token-level retrieval
  at each inference step to compute a semi-parametric mixture distribution and
  identify promising span continuations in a corpus. It then uses an approximate
  speculative decoding procedure that accepts a prefix of the retrieved span or
  generates a new token. NEST significantly enhances the generation quality and
  attribution rate of the base LM across a variety of knowledge-intensive tasks,
  surpassing the conventional kNN-LM method and performing competitively with
  in-context retrieval augmentation. In addition, NEST substantially improves the
  generation speed, achieving a 1.8x speedup in inference time when applied to
  Llama-2-Chat 70B.
paper_authors._id: 665887f61c264957e5a76381
paper_authors.name: Minghan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:52:17.203Z
paper_authors._id: 61340b5e2d28e0a2db921ee3
paper_authors.avatarUrl: /avatars/5c1be8985ef2e68ca2508f9c1a9ff1f9.svg
paper_authors.fullname: Minghan Li
paper_authors.type: user
paper_authors.user: alexlimh
paper_authors._id: 665887f61c264957e5a76382
paper_authors.name: Xilun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:52:34.754Z
paper_authors._id: 62ffccd18813dd8628172c6c
paper_authors.avatarUrl: /avatars/62741be5967a74758a2b7914aa842ec8.svg
paper_authors.fullname: Xilun Chen
paper_authors.type: user
paper_authors.user: ccsasuke
paper_authors._id: 665887f61c264957e5a76383
paper_authors.name: Ari Holtzman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:53:01.346Z
paper_authors._id: 6508c81db44445e9b36f3a1e
paper_authors.avatarUrl: /avatars/7d4b257398c601b482b9655f046021f3.svg
paper_authors.fullname: Ari Holtzman
paper_authors.type: user
paper_authors.user: ariholtzman
paper_authors._id: 665887f61c264957e5a76384
paper_authors.name: Beidi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:53:07.097Z
paper_authors._id: 64b732f832403871593e082c
paper_authors.avatarUrl: /avatars/dd21932b0c167131ee7545a622c46c3c.svg
paper_authors.fullname: Beidi Chen
paper_authors.type: user
paper_authors.user: beidic
paper_authors._id: 665887f61c264957e5a76385
paper_authors.name: Jimmy Lin
paper_authors._id: 665887f61c264957e5a76386
paper_authors.name: Wen-tau Yih
paper_authors._id: 665887f61c264957e5a76387
paper_authors.name: Xi Victoria Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-31T10:54:51.458Z
paper_authors._id: 61df349e34b589fbaf2e93c4
paper_authors.avatarUrl: /avatars/9dd7776d3b610d78e40c2e178ab2f112.svg
paper_authors.fullname: Victoria Lin
paper_authors.type: user
paper_authors.user: VictoriaLinML
paper_title: Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
paper_summary: Large language models (LLMs) often hallucinate and lack the ability to
  provide attribution for their generations. Semi-parametric LMs, such as kNN-LM,
  approach these limitations by refining the output of an LM for a given prompt
  using its nearest neighbor matches in a non-parametric data store. However,
  these models often exhibit slow inference speeds and produce non-fluent texts.
  In this paper, we introduce Nearest Neighbor Speculative Decoding (NEST), a
  novel semi-parametric language modeling approach that is capable of
  incorporating real-world text spans of arbitrary length into the LM generations
  and providing attribution to their sources. NEST performs token-level retrieval
  at each inference step to compute a semi-parametric mixture distribution and
  identify promising span continuations in a corpus. It then uses an approximate
  speculative decoding procedure that accepts a prefix of the retrieved span or
  generates a new token. NEST significantly enhances the generation quality and
  attribution rate of the base LM across a variety of knowledge-intensive tasks,
  surpassing the conventional kNN-LM method and performing competitively with
  in-context retrieval augmentation. In addition, NEST substantially improves the
  generation speed, achieving a 1.8x speedup in inference time when applied to
  Llama-2-Chat 70B.
paper_ai_summary: Nearest Neighbor Speculative Decoding enhances language model generation quality, attribution, and speed through token-level retrieval and speculative decoding.
paper_ai_keywords: semi-parametric LMs kNN-LM Nearest Neighbor Speculative Decoding token-level retrieval semi-parametric mixture distribution speculative decoding generation quality attribution rate knowledge-intensive tasks in-context retrieval augmentation inference speed