title: DoLa: Decoding by Contrasting Layers Improves Factuality in Large
    Language Models
summary: Despite their impressive capabilities, large language models (LLMs) are prone
  to hallucinations, i.e., generating content that deviates from facts seen
  during pretraining. We propose a simple decoding strategy for reducing
  hallucinations with pretrained LLMs that does not require conditioning on
  retrieved external knowledge nor additional fine-tuning. Our approach obtains
  the next-token distribution by contrasting the differences in logits obtained
  from projecting the later layers versus earlier layers to the vocabulary space,
  exploiting the fact that factual knowledge in an LLMs has generally been shown
  to be localized to particular transformer layers. We find that this Decoding by
  Contrasting Layers (DoLa) approach is able to better surface factual knowledge
  and reduce the generation of incorrect facts. DoLa consistently improves the
  truthfulness across multiple choices tasks and open-ended generation tasks, for
  example improving the performance of LLaMA family models on TruthfulQA by
  12-17% absolute points, demonstrating its potential in making LLMs reliably
  generate truthful facts.
paper_authors._id: 64fa833ec367f7b1caba956d
paper_authors.name: Yung-Sung Chuang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-09-08T03:40:31.909Z
paper_authors._id: 5df84571da6d0311fd3d5407
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1650651305661-5df84571da6d0311fd3d5407.png
paper_authors.fullname: Yung-Sung Chuang
paper_authors.type: user
paper_authors.user: voidism
paper_authors._id: 64fa833ec367f7b1caba956e
paper_authors.name: Yujia Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:28:10.177Z
paper_authors._id: 62cdb0bc3529c21a2283bc19
paper_authors.avatarUrl: /avatars/414dccde596e4a401163814b1f3ca156.svg
paper_authors.fullname: Yujia Xie
paper_authors.type: user
paper_authors.user: xieyujia
paper_authors._id: 64fa833ec367f7b1caba956f
paper_authors.name: Hongyin Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:28:49.522Z
paper_authors._id: 62bc495016897fa76cdcda18
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656506661693-noauth.jpeg
paper_authors.fullname: Hongyin Luo
paper_authors.type: user
paper_authors.user: luohy
paper_authors._id: 64fa833ec367f7b1caba9570
paper_authors.name: Yoon Kim
paper_authors._id: 64fa833ec367f7b1caba9571
paper_authors.name: James Glass
paper_authors._id: 64fa833ec367f7b1caba9572
paper_authors.name: Pengcheng He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:29:54.895Z
paper_authors._id: 5efd09cf49ed724c8a135868
paper_authors.avatarUrl: /avatars/af12bc94657979677a9f26183f0c9727.svg
paper_authors.fullname: Pengcheng He
paper_authors.type: user
paper_authors.user: DeBERTa
paper_title: DoLa: Decoding by Contrasting Layers Improves Factuality in Large
    Language Models
paper_summary: Despite their impressive capabilities, large language models (LLMs) are prone
  to hallucinations, i.e., generating content that deviates from facts seen
  during pretraining. We propose a simple decoding strategy for reducing
  hallucinations with pretrained LLMs that does not require conditioning on
  retrieved external knowledge nor additional fine-tuning. Our approach obtains
  the next-token distribution by contrasting the differences in logits obtained
  from projecting the later layers versus earlier layers to the vocabulary space,
  exploiting the fact that factual knowledge in an LLMs has generally been shown
  to be localized to particular transformer layers. We find that this Decoding by
  Contrasting Layers (DoLa) approach is able to better surface factual knowledge
  and reduce the generation of incorrect facts. DoLa consistently improves the
  truthfulness across multiple choices tasks and open-ended generation tasks, for
  example improving the performance of LLaMA family models on TruthfulQA by
  12-17% absolute points, demonstrating its potential in making LLMs reliably
  generate truthful facts.
paper_ai_summary: A decoding strategy that contrasts logits from different transformer layers reduces hallucinations in large language models without external knowledge or fine-tuning.
paper_ai_keywords: large language models LLMs hallucinations decoding strategy logits vocabulary space transformer layers factual knowledge Decoding by Contrasting Layers DoLa TruthfulQA