title: Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
summary: Large language models (LLMs) exhibit hallucinations (i.e., unfaithful or
  nonsensical information) when serving as AI assistants in various domains.
  Since hallucinations always come with truthful content in the LLM responses,
  previous factuality alignment methods that conduct response-level preference
  learning inevitably introduced noises during training. Therefore, this paper
  proposes a fine-grained factuality alignment method based on Direct Preference
  Optimization (DPO), called Mask-DPO. Incorporating sentence-level factuality as
  mask signals, Mask-DPO only learns from factually correct sentences in the
  preferred samples and prevents the penalty on factual contents in the not
  preferred samples, which resolves the ambiguity in the preference learning.
  Extensive experimental results demonstrate that Mask-DPO can significantly
  improve the factuality of LLMs responses to questions from both in-domain and
  out-of-domain datasets, although these questions and their corresponding topics
  are unseen during training. Only trained on the ANAH train set, the score of
  Llama3.1-8B-Instruct on the ANAH test set is improved from 49.19% to 77.53%,
  even surpassing the score of Llama3.1-70B-Instruct (53.44%), while its
  FactScore on the out-of-domain Biography dataset is also improved from 30.29%
  to 39.39%. We further study the generalization property of Mask-DPO using
  different training sample scaling strategies and find that scaling the number
  of topics in the dataset is more effective than the number of questions. We
  provide a hypothesis of what factual alignment is doing with LLMs, on the
  implication of this phenomenon, and conduct proof-of-concept experiments to
  verify it. We hope the method and the findings pave the way for future research
  on scaling factuality alignment.
paper_authors._id: 67c7c3ce6f68759bf368533c
paper_authors.name: Yuzhe Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:11:38.988Z
paper_authors._id: 6601196cc91ba4c08ad6e270
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6601196cc91ba4c08ad6e270/X2YPNzUOQXBz5Gv-xR9LW.jpeg
paper_authors.fullname: yuzhe gu
paper_authors.type: user
paper_authors.user: vanilla1116
paper_authors._id: 67c7c3ce6f68759bf368533d
paper_authors.name: Wenwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:32:26.106Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 67c7c3ce6f68759bf368533e
paper_authors.name: Chengqi Lyu
paper_authors._id: 67c7c3ce6f68759bf368533f
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-05T10:32:37.093Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 67c7c3ce6f68759bf3685340
paper_authors.name: Kai Chen
paper_title: Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs
paper_summary: Large language models (LLMs) exhibit hallucinations (i.e., unfaithful or
  nonsensical information) when serving as AI assistants in various domains.
  Since hallucinations always come with truthful content in the LLM responses,
  previous factuality alignment methods that conduct response-level preference
  learning inevitably introduced noises during training. Therefore, this paper
  proposes a fine-grained factuality alignment method based on Direct Preference
  Optimization (DPO), called Mask-DPO. Incorporating sentence-level factuality as
  mask signals, Mask-DPO only learns from factually correct sentences in the
  preferred samples and prevents the penalty on factual contents in the not
  preferred samples, which resolves the ambiguity in the preference learning.
  Extensive experimental results demonstrate that Mask-DPO can significantly
  improve the factuality of LLMs responses to questions from both in-domain and
  out-of-domain datasets, although these questions and their corresponding topics
  are unseen during training. Only trained on the ANAH train set, the score of
  Llama3.1-8B-Instruct on the ANAH test set is improved from 49.19% to 77.53%,
  even surpassing the score of Llama3.1-70B-Instruct (53.44%), while its
  FactScore on the out-of-domain Biography dataset is also improved from 30.29%
  to 39.39%. We further study the generalization property of Mask-DPO using
  different training sample scaling strategies and find that scaling the number
  of topics in the dataset is more effective than the number of questions. We
  provide a hypothesis of what factual alignment is doing with LLMs, on the
  implication of this phenomenon, and conduct proof-of-concept experiments to
  verify it. We hope the method and the findings pave the way for future research
  on scaling factuality alignment.
paper_ai_summary: Mask-DPO, a fine-grained factuality alignment method based on Direct Preference Optimization, improves the factuality of LLM responses by learning from factually correct sentences and generalizes to unseen questions and topics.
paper_ai_keywords: Large language models (LLMs) hallucinations Direct Preference Optimization (DPO) sentence-level factuality Mask-DPO factuality alignment