title: Fine-tuning Language Models for Factuality
summary: The fluency and creativity of large pre-trained language models (LLMs) have
  led to their widespread use, sometimes even as a replacement for traditional
  search engines. Yet language models are prone to making convincing but
  factually inaccurate claims, often referred to as 'hallucinations.' These
  errors can inadvertently spread misinformation or harmfully perpetuate
  misconceptions. Further, manual fact-checking of model responses is a
  time-consuming process, making human factuality labels expensive to acquire. In
  this work, we fine-tune language models to be more factual, without human
  labeling and targeting more open-ended generation settings than past work. We
  leverage two key recent innovations in NLP to do so. First, several recent
  works have proposed methods for judging the factuality of open-ended text by
  measuring consistency with an external knowledge base or simply a large model's
  confidence scores. Second, the direct preference optimization algorithm enables
  straightforward fine-tuning of language models on objectives other than
  supervised imitation, using a preference ranking over possible model responses.
  We show that learning from automatically generated factuality preference
  rankings, generated either through existing retrieval systems or our novel
  retrieval-free approach, significantly improves the factuality (percent of
  generated claims that are correct) of Llama-2 on held-out topics compared with
  RLHF or decoding strategies targeted at factuality. At 7B scale, compared to
  Llama-2-chat, we observe 58% and 40% reduction in factual error rate when
  generating biographies and answering medical questions, respectively.
paper_authors._id: 6554422a78594ac5478b3353
paper_authors.name: Katherine Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:16:30.852Z
paper_authors._id: 64b5af69c3c3121795db760f
paper_authors.avatarUrl: /avatars/5b3cb57370f3546a9bbfed5b8c8d5ca7.svg
paper_authors.fullname: Katherine Tian
paper_authors.type: user
paper_authors.user: ktian
paper_authors._id: 6554422a78594ac5478b3354
paper_authors.name: Eric Mitchell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:16:38.062Z
paper_authors._id: 6514e9cbfc2301d62d7bf2ce
paper_authors.avatarUrl: /avatars/eba6bd6e2840d7b5dd735e6662020d0c.svg
paper_authors.fullname: Eric mitchell
paper_authors.type: user
paper_authors.user: Pain777
paper_authors._id: 6554422a78594ac5478b3355
paper_authors.name: Huaxiu Yao
paper_authors._id: 6554422a78594ac5478b3356
paper_authors.name: Christopher D. Manning
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:17:07.821Z
paper_authors._id: 5f555a1f8bf55658acfed19e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1628695407761-5f555a1f8bf55658acfed19e.jpeg
paper_authors.fullname: Christopher Manning
paper_authors.type: user
paper_authors.user: manning
paper_authors._id: 6554422a78594ac5478b3357
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-15T10:17:14.217Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_title: Fine-tuning Language Models for Factuality
paper_summary: The fluency and creativity of large pre-trained language models (LLMs) have
  led to their widespread use, sometimes even as a replacement for traditional
  search engines. Yet language models are prone to making convincing but
  factually inaccurate claims, often referred to as 'hallucinations.' These
  errors can inadvertently spread misinformation or harmfully perpetuate
  misconceptions. Further, manual fact-checking of model responses is a
  time-consuming process, making human factuality labels expensive to acquire. In
  this work, we fine-tune language models to be more factual, without human
  labeling and targeting more open-ended generation settings than past work. We
  leverage two key recent innovations in NLP to do so. First, several recent
  works have proposed methods for judging the factuality of open-ended text by
  measuring consistency with an external knowledge base or simply a large model's
  confidence scores. Second, the direct preference optimization algorithm enables
  straightforward fine-tuning of language models on objectives other than
  supervised imitation, using a preference ranking over possible model responses.
  We show that learning from automatically generated factuality preference
  rankings, generated either through existing retrieval systems or our novel
  retrieval-free approach, significantly improves the factuality (percent of
  generated claims that are correct) of Llama-2 on held-out topics compared with
  RLHF or decoding strategies targeted at factuality. At 7B scale, compared to
  Llama-2-chat, we observe 58% and 40% reduction in factual error rate when
  generating biographies and answering medical questions, respectively.
paper_ai_keywords: large pre-trained language models (LLMs) hallucinations manual fact-checking fine-tuning open-ended generation settings external knowledge base direct preference optimization algorithm RLHF (Reinforcement Learning from Human Feedback) held-out topics factual error rate Llama-2 Llama-2-chat biographies medical questions