title: Teaching Large Language Models to Maintain Contextual Faithfulness via
    Synthetic Tasks and Reinforcement Learning
summary: Teaching large language models (LLMs) to be faithful in the provided context
  is crucial for building reliable information-seeking systems. Therefore, we
  propose a systematic framework, CANOE, to improve the faithfulness of LLMs in
  both short-form and long-form generation tasks without human annotations.
  Specifically, we first synthesize short-form question-answering (QA) data with
  four diverse tasks to construct high-quality and easily verifiable training
  data without human annotation. Also, we propose Dual-GRPO, a rule-based
  reinforcement learning method that includes three tailored rule-based rewards
  derived from synthesized short-form QA data, while simultaneously optimizing
  both short-form and long-form response generation. Notably, Dual-GRPO
  eliminates the need to manually label preference data to train reward models
  and avoids over-optimizing short-form generation when relying only on the
  synthesized short-form QA data. Experimental results show that CANOE greatly
  improves the faithfulness of LLMs across 11 different downstream tasks, even
  outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.
paper_authors._id: 6833cb27e10e89e250a6d9ae
paper_authors.name: Shuzheng Si
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:55:07.428Z
paper_authors._id: 637c99bbfe115289cfedfb44
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637c99bbfe115289cfedfb44/344NN9KKF_XXTlVYaGaMW.png
paper_authors.fullname: ssz
paper_authors.type: user
paper_authors.user: ssz1111
paper_authors._id: 6833cb27e10e89e250a6d9af
paper_authors.name: Haozhe Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:10:14.872Z
paper_authors._id: 63ff09a02098b9ad105a09f6
paper_authors.avatarUrl: /avatars/4409ca5d320050cf4c3df05962c7ff58.svg
paper_authors.fullname: Hans Zhao
paper_authors.type: user
paper_authors.user: BleachNick
paper_authors._id: 6833cb27e10e89e250a6d9b0
paper_authors.name: Cheng Gao
paper_authors._id: 6833cb27e10e89e250a6d9b1
paper_authors.name: Yuzhuo Bai
paper_authors._id: 6833cb27e10e89e250a6d9b2
paper_authors.name: Zhitong Wang
paper_authors._id: 6833cb27e10e89e250a6d9b3
paper_authors.name: Bofei Gao
paper_authors._id: 6833cb27e10e89e250a6d9b4
paper_authors.name: Kangyang Luo
paper_authors._id: 6833cb27e10e89e250a6d9b5
paper_authors.name: Wenhao Li
paper_authors._id: 6833cb27e10e89e250a6d9b6
paper_authors.name: Yufei Huang
paper_authors._id: 6833cb27e10e89e250a6d9b7
paper_authors.name: Gang Chen
paper_authors._id: 6833cb27e10e89e250a6d9b8
paper_authors.name: Fanchao Qi
paper_authors._id: 6833cb27e10e89e250a6d9b9
paper_authors.name: Minjia Zhang
paper_authors._id: 6833cb27e10e89e250a6d9ba
paper_authors.name: Baobao Chang
paper_authors._id: 6833cb27e10e89e250a6d9bb
paper_authors.name: Maosong Sun
paper_title: Teaching Large Language Models to Maintain Contextual Faithfulness via
    Synthetic Tasks and Reinforcement Learning
paper_summary: Teaching large language models (LLMs) to be faithful in the provided context
  is crucial for building reliable information-seeking systems. Therefore, we
  propose a systematic framework, CANOE, to improve the faithfulness of LLMs in
  both short-form and long-form generation tasks without human annotations.
  Specifically, we first synthesize short-form question-answering (QA) data with
  four diverse tasks to construct high-quality and easily verifiable training
  data without human annotation. Also, we propose Dual-GRPO, a rule-based
  reinforcement learning method that includes three tailored rule-based rewards
  derived from synthesized short-form QA data, while simultaneously optimizing
  both short-form and long-form response generation. Notably, Dual-GRPO
  eliminates the need to manually label preference data to train reward models
  and avoids over-optimizing short-form generation when relying only on the
  synthesized short-form QA data. Experimental results show that CANOE greatly
  improves the faithfulness of LLMs across 11 different downstream tasks, even
  outperforming the most advanced LLMs, e.g., GPT-4o and OpenAI o1.
paper_ai_summary: CANOE improves LLM faithfulness in generation tasks using synthetic QA data and Dual-GRPO reinforcement learning without human annotations.
paper_ai_keywords: teaching large language models faithfulness context CANOE short-form generation long-form generation question-answering Dual-GRPO rule-based reinforcement learning preference data reward models