Fine-tuning Language Models for Factuality
The fluency and creativity of large pre-trained language models (LLMs) have
led to their widespread use, sometimes even as a replacement for traditional
search engines. Yet language models are prone to making convincing but
factually inaccurate claims, often referred to as 'hallucinations.' These
errors can inadvertently spread misinformation or harmfully perpetuate
misconceptions. Further, manual fact-checking of model responses is a
time-consuming process, making human factuality labels expensive to acquire. In
this work, we fine-tune language models to be more factual, without human
labeling and targeting more open-ended generation settings than past work. We
leverage two key recent innovations in NLP to do so. First, several recent
works have proposed methods for judging the factuality of open-ended text by
measuring consistency with an external knowledge base or simply a large model's
confidence scores. Second, the direct preference optimization algorithm enables
straightforward fine-tuning of language models on objectives other than
supervised imitation, using a preference ranking over possible model responses.
We show that learning from automatically generated factuality preference
rankings, generated either through existing retrieval systems or our novel
retrieval-free approach, significantly improves the factuality (percent of
generated claims that are correct) of Llama-2 on held-out topics compared with
RLHF or decoding strategies targeted at factuality. At 7B scale, compared to
Llama-2-chat, we observe 58% and 40% reduction in factual error rate when
generating biographies and answering medical questions, respectively.
6554422a78594ac5478b3353
Katherine Tian
admin_assigned
2023-11-15T10:16:30.852Z
64b5af69c3c3121795db760f
/avatars/5b3cb57370f3546a9bbfed5b8c8d5ca7.svg
Katherine Tian
user
ktian
6554422a78594ac5478b3354
Eric Mitchell
admin_assigned
2023-11-15T10:16:38.062Z
6514e9cbfc2301d62d7bf2ce
/avatars/eba6bd6e2840d7b5dd735e6662020d0c.svg
Eric mitchell
user
Pain777
6554422a78594ac5478b3355
Huaxiu Yao
6554422a78594ac5478b3356
Christopher D. Manning
admin_assigned
2023-11-15T10:17:07.821Z
5f555a1f8bf55658acfed19e
https://cdn-avatars.huggingface.co/v1/production/uploads/1628695407761-5f555a1f8bf55658acfed19e.jpeg
Christopher Manning
user
manning
6554422a78594ac5478b3357
Chelsea Finn
admin_assigned
2023-11-15T10:17:14.217Z
64ac22a9193f0a807deb673d
/avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
Chelsea Finn
user
cbfinn
Fine-tuning Language Models for Factuality
The fluency and creativity of large pre-trained language models (LLMs) have
led to their widespread use, sometimes even as a replacement for traditional
search engines. Yet language models are prone to making convincing but
factually inaccurate claims, often referred to as 'hallucinations.' These
errors can inadvertently spread misinformation or harmfully perpetuate
misconceptions. Further, manual fact-checking of model responses is a
time-consuming process, making human factuality labels expensive to acquire. In
this work, we fine-tune language models to be more factual, without human
labeling and targeting more open-ended generation settings than past work. We
leverage two key recent innovations in NLP to do so. First, several recent
works have proposed methods for judging the factuality of open-ended text by
measuring consistency with an external knowledge base or simply a large model's
confidence scores. Second, the direct preference optimization algorithm enables
straightforward fine-tuning of language models on objectives other than
supervised imitation, using a preference ranking over possible model responses.
We show that learning from automatically generated factuality preference
rankings, generated either through existing retrieval systems or our novel
retrieval-free approach, significantly improves the factuality (percent of
generated claims that are correct) of Llama-2 on held-out topics compared with
RLHF or decoding strategies targeted at factuality. At 7B scale, compared to
Llama-2-chat, we observe 58% and 40% reduction in factual error rate when
generating biographies and answering medical questions, respectively.
Fine-tuning language models using automatically generated factuality preference rankings improves their factual accuracy without human labeling.
large pre-trained language models LLMs language models factually inaccurate claims hallucinations manual fact-checking open-ended generation external knowledge base large model's confidence scores direct preference optimization RLHF decoding strategies factual error rate biographies medical questions