title: Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite
    Learning
summary: Recent studies have identified one aggravating factor of LLM hallucinations
  as the knowledge inconsistency between pre-training and fine-tuning, where
  unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong
  outputs. In this paper, we propose a novel fine-tuning strategy called
  Prereq-Tune to address this knowledge inconsistency and reduce hallucinations.
  Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge,
  so the model learns only the task skills without being impacted by the
  knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional
  prerequisite learning stage to learn the necessary knowledge for SFT, allowing
  subsequent SFT to focus only on task skills. Prereq-Tune can also be combined
  with fictitious synthetic data to enhance the grounding of LLM outputs to their
  internal knowledge. Experiments show that Prereq-Tune outperforms existing
  baselines in improving LLM's factuality across short QA and long-form
  generation tasks. It also opens new possibilities for knowledge-controlled
  generation in LLMs. Our code is available at
  https://github.com/UCSB-NLP-Chang/Prereq_tune.git.
paper_authors._id: 671f0d24941d8e30b6a43180
paper_authors.name: Yujian Liu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-28T04:09:03.357Z
paper_authors._id: 615025173e89795099b6fcd9
paper_authors.avatarUrl: /avatars/9e2ebea7c5c2d38c95e68af13cc9e382.svg
paper_authors.fullname: Yujian Liu
paper_authors.type: user
paper_authors.user: yujianll
paper_authors._id: 671f0d24941d8e30b6a43181
paper_authors.name: Shiyu Chang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-28T13:57:57.969Z
paper_authors._id: 6696d191d79ce5b27da89c95
paper_authors.avatarUrl: /avatars/820c2631c1715a50edb628493656b64f.svg
paper_authors.fullname: Shiyu Chang
paper_authors.type: user
paper_authors.user: code-terminator
paper_authors._id: 671f0d24941d8e30b6a43182
paper_authors.name: Tommi Jaakkola
paper_authors._id: 671f0d24941d8e30b6a43183
paper_authors.name: Yang Zhang
paper_title: Fictitious Synthetic Data Can Improve LLM Factuality via Prerequisite
    Learning
paper_summary: Recent studies have identified one aggravating factor of LLM hallucinations
  as the knowledge inconsistency between pre-training and fine-tuning, where
  unfamiliar fine-tuning data mislead the LLM to fabricate plausible but wrong
  outputs. In this paper, we propose a novel fine-tuning strategy called
  Prereq-Tune to address this knowledge inconsistency and reduce hallucinations.
  Fundamentally, Prereq-Tune disentangles the learning of skills and knowledge,
  so the model learns only the task skills without being impacted by the
  knowledge inconsistency. To achieve this, Prereq-Tune introduces an additional
  prerequisite learning stage to learn the necessary knowledge for SFT, allowing
  subsequent SFT to focus only on task skills. Prereq-Tune can also be combined
  with fictitious synthetic data to enhance the grounding of LLM outputs to their
  internal knowledge. Experiments show that Prereq-Tune outperforms existing
  baselines in improving LLM's factuality across short QA and long-form
  generation tasks. It also opens new possibilities for knowledge-controlled
  generation in LLMs. Our code is available at
  https://github.com/UCSB-NLP-Chang/Prereq_tune.git.
paper_ai_keywords: LLMs (Language Models) hallucinations pre-training fine-tuning knowledge inconsistency SFT (Supervised Fine-Tuning) Prereq-Tune prerequisite learning stage synthetic data grounding factuality long-form generation tasks knowledge-controlled generation