title: FLAME: Factuality-Aware Alignment for Large Language Models
summary: Alignment is a standard procedure to fine-tune pre-trained large language
  models (LLMs) to follow natural language instructions and serve as helpful AI
  assistants. We have observed, however, that the conventional alignment process
  fails to enhance the factual accuracy of LLMs, and often leads to the
  generation of more false facts (i.e. hallucination). In this paper, we study
  how to make the LLM alignment process more factual, by first identifying
  factors that lead to hallucination in both alignment steps:\ supervised
  fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that
  training the LLM on new knowledge or unfamiliar texts can encourage
  hallucination. This makes SFT less factual as it trains on human labeled data
  that may be novel to the LLM. Furthermore, reward functions used in standard RL
  can also encourage hallucination, because it guides the LLM to provide more
  helpful responses on a diverse set of instructions, often preferring longer and
  more detailed responses. Based on these observations, we propose
  factuality-aware alignment, comprised of factuality-aware SFT and
  factuality-aware RL through direct preference optimization. Experiments show
  that our proposed factuality-aware alignment guides LLMs to output more factual
  responses while maintaining instruction-following capability.
paper_authors._id: 663452b01a86dae982449bed
paper_authors.name: Sheng-Chieh Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:50:59.120Z
paper_authors._id: 616deb42cfb146b4ed9ae027
paper_authors.avatarUrl: /avatars/b847823e3f484f0072e0adb792a95ac3.svg
paper_authors.fullname: Sheng-Chieh Lin
paper_authors.type: user
paper_authors.user: jacklin
paper_authors._id: 663452b01a86dae982449bee
paper_authors.name: Luyu Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:51:10.278Z
paper_authors._id: 6009aa9126e23ab5edb47afd
paper_authors.avatarUrl: /avatars/f8afa9cf648227ad5817323451ede378.svg
paper_authors.fullname: Luyu Gao
paper_authors.type: user
paper_authors.user: Luyu
paper_authors._id: 663452b01a86dae982449bef
paper_authors.name: Barlas Oguz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:51:27.583Z
paper_authors._id: 6317cb5b212fce5a3cd97227
paper_authors.avatarUrl: /avatars/c2b4eefa7c17cdf2299c07466c361ab0.svg
paper_authors.fullname: Barlas Oguz
paper_authors.type: user
paper_authors.user: barlaso
paper_authors._id: 663452b01a86dae982449bf0
paper_authors.name: Wenhan Xiong
paper_authors._id: 663452b01a86dae982449bf1
paper_authors.name: Jimmy Lin
paper_authors._id: 663452b01a86dae982449bf2
paper_authors.name: Wen-tau Yih
paper_authors._id: 663452b01a86dae982449bf3
paper_authors.name: Xilun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-03T07:53:17.830Z
paper_authors._id: 62ffccd18813dd8628172c6c
paper_authors.avatarUrl: /avatars/62741be5967a74758a2b7914aa842ec8.svg
paper_authors.fullname: Xilun Chen
paper_authors.type: user
paper_authors.user: ccsasuke
paper_title: FLAME: Factuality-Aware Alignment for Large Language Models
paper_summary: Alignment is a standard procedure to fine-tune pre-trained large language
  models (LLMs) to follow natural language instructions and serve as helpful AI
  assistants. We have observed, however, that the conventional alignment process
  fails to enhance the factual accuracy of LLMs, and often leads to the
  generation of more false facts (i.e. hallucination). In this paper, we study
  how to make the LLM alignment process more factual, by first identifying
  factors that lead to hallucination in both alignment steps:\ supervised
  fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that
  training the LLM on new knowledge or unfamiliar texts can encourage
  hallucination. This makes SFT less factual as it trains on human labeled data
  that may be novel to the LLM. Furthermore, reward functions used in standard RL
  can also encourage hallucination, because it guides the LLM to provide more
  helpful responses on a diverse set of instructions, often preferring longer and
  more detailed responses. Based on these observations, we propose
  factuality-aware alignment, comprised of factuality-aware SFT and
  factuality-aware RL through direct preference optimization. Experiments show
  that our proposed factuality-aware alignment guides LLMs to output more factual
  responses while maintaining instruction-following capability.
paper_ai_keywords: pre-trained large language models alignment process hallucination vocal fine-tuning reinforcement learning factuality-aware alignment factuality-aware SFT factuality-aware RL direct preference optimization