FLAME: Factuality-Aware Alignment for Large Language Models
Alignment is a standard procedure to fine-tune pre-trained large language
models (LLMs) to follow natural language instructions and serve as helpful AI
assistants. We have observed, however, that the conventional alignment process
fails to enhance the factual accuracy of LLMs, and often leads to the
generation of more false facts (i.e. hallucination). In this paper, we study
how to make the LLM alignment process more factual, by first identifying
factors that lead to hallucination in both alignment steps:\ supervised
fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that
training the LLM on new knowledge or unfamiliar texts can encourage
hallucination. This makes SFT less factual as it trains on human labeled data
that may be novel to the LLM. Furthermore, reward functions used in standard RL
can also encourage hallucination, because it guides the LLM to provide more
helpful responses on a diverse set of instructions, often preferring longer and
more detailed responses. Based on these observations, we propose
factuality-aware alignment, comprised of factuality-aware SFT and
factuality-aware RL through direct preference optimization. Experiments show
that our proposed factuality-aware alignment guides LLMs to output more factual
responses while maintaining instruction-following capability.
663452b01a86dae982449bed
Sheng-Chieh Lin
admin_assigned
2024-05-03T07:50:59.120Z
616deb42cfb146b4ed9ae027
/avatars/b847823e3f484f0072e0adb792a95ac3.svg
Sheng-Chieh Lin
user
jacklin
663452b01a86dae982449bee
Luyu Gao
admin_assigned
2024-05-03T07:51:10.278Z
6009aa9126e23ab5edb47afd
/avatars/f8afa9cf648227ad5817323451ede378.svg
Luyu Gao
user
Luyu
663452b01a86dae982449bef
Barlas Oguz
admin_assigned
2024-05-03T07:51:27.583Z
6317cb5b212fce5a3cd97227
/avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
Barlas Oguz
user
barlaso
663452b01a86dae982449bf0
Wenhan Xiong
663452b01a86dae982449bf1
Jimmy Lin
663452b01a86dae982449bf2
Wen-tau Yih
663452b01a86dae982449bf3
Xilun Chen
admin_assigned
2024-05-03T07:53:17.830Z
62ffccd18813dd8628172c6c
/avatars/62741be5967a74758a2b7914aa842ec8.svg
Xilun Chen
user
ccsasuke
FLAME: Factuality-Aware Alignment for Large Language Models
Alignment is a standard procedure to fine-tune pre-trained large language
models (LLMs) to follow natural language instructions and serve as helpful AI
assistants. We have observed, however, that the conventional alignment process
fails to enhance the factual accuracy of LLMs, and often leads to the
generation of more false facts (i.e. hallucination). In this paper, we study
how to make the LLM alignment process more factual, by first identifying
factors that lead to hallucination in both alignment steps:\ supervised
fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that
training the LLM on new knowledge or unfamiliar texts can encourage
hallucination. This makes SFT less factual as it trains on human labeled data
that may be novel to the LLM. Furthermore, reward functions used in standard RL
can also encourage hallucination, because it guides the LLM to provide more
helpful responses on a diverse set of instructions, often preferring longer and
more detailed responses. Based on these observations, we propose
factuality-aware alignment, comprised of factuality-aware SFT and
factuality-aware RL through direct preference optimization. Experiments show
that our proposed factuality-aware alignment guides LLMs to output more factual
responses while maintaining instruction-following capability.
Proposes factuality-aware alignment methods for pre-trained language models to enhance factual accuracy during supervised fine-tuning and reinforcement learning, reducing hallucination.
large language models LLMS supervised fine-tuning SFT reinforcement learning RL hallucination factuality-aware alignment direct preference optimization