title: LongReward: Improving Long-context Large Language Models with AI
    Feedback
summary: Though significant advancements have been achieved in developing long-context
  large language models (LLMs), the compromised quality of LLM-synthesized data
  for supervised fine-tuning (SFT) often affects the long-context performance of
  SFT models and leads to inherent limitations. In principle, reinforcement
  learning (RL) with appropriate reward signals can further enhance models'
  capacities. However, how to obtain reliable rewards in long-context scenarios
  remains unexplored. To this end, we propose LongReward, a novel method that
  utilizes an off-the-shelf LLM to provide rewards for long-context model
  responses from four human-valued dimensions: helpfulness, logicality,
  faithfulness, and completeness, each with a carefully designed assessment
  pipeline. By combining LongReward and offline RL algorithm DPO, we are able to
  effectively improve long-context SFT models. Our experiments indicate that
  LongReward not only significantly improves models' long-context performance but
  also enhances their ability to follow short instructions. We also find that
  long-context DPO with LongReward and conventional short-context DPO can be used
  together without hurting either one's performance.
paper_authors._id: 67204419dc8f6146490a8253
paper_authors.name: Jiajie Zhang
paper_authors._id: 67204419dc8f6146490a8254
paper_authors.name: Zhongni Hou
paper_authors._id: 67204419dc8f6146490a8255
paper_authors.name: Xin Lv
paper_authors._id: 67204419dc8f6146490a8256
paper_authors.name: Shulin Cao
paper_authors._id: 67204419dc8f6146490a8257
paper_authors.name: Zhenyu Hou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T09:22:00.783Z
paper_authors._id: 62b196646d3d059f40c3df19
paper_authors.avatarUrl: /avatars/dbddf54ae949437223f3a438d30ef653.svg
paper_authors.fullname: Zhenyu Hou
paper_authors.type: user
paper_authors.user: think2try
paper_authors._id: 67204419dc8f6146490a8258
paper_authors.name: Yilin Niu
paper_authors._id: 67204419dc8f6146490a8259
paper_authors.name: Lei Hou
paper_authors._id: 67204419dc8f6146490a825a
paper_authors.name: Yuxiao Dong
paper_authors._id: 67204419dc8f6146490a825b
paper_authors.name: Ling Feng
paper_authors._id: 67204419dc8f6146490a825c
paper_authors.name: Juanzi Li
paper_title: LongReward: Improving Long-context Large Language Models with AI
    Feedback
paper_summary: Though significant advancements have been achieved in developing long-context
  large language models (LLMs), the compromised quality of LLM-synthesized data
  for supervised fine-tuning (SFT) often affects the long-context performance of
  SFT models and leads to inherent limitations. In principle, reinforcement
  learning (RL) with appropriate reward signals can further enhance models'
  capacities. However, how to obtain reliable rewards in long-context scenarios
  remains unexplored. To this end, we propose LongReward, a novel method that
  utilizes an off-the-shelf LLM to provide rewards for long-context model
  responses from four human-valued dimensions: helpfulness, logicality,
  faithfulness, and completeness, each with a carefully designed assessment
  pipeline. By combining LongReward and offline RL algorithm DPO, we are able to
  effectively improve long-context SFT models. Our experiments indicate that
  LongReward not only significantly improves models' long-context performance but
  also enhances their ability to follow short instructions. We also find that
  long-context DPO with LongReward and conventional short-context DPO can be used
  together without hurting either one's performance.
paper_ai_summary: A novel reward-based method called LongReward enhances long-context performance of supervised fine-tuned models by leveraging human-valued criteria and an off-the-shelf LLM, and it can be combined with offline RL algorithms like DPO effectively.
paper_ai_keywords: long-context large language models supervised fine-tuning reinforcement learning reward signals LongReward assessment pipeline offline RL algorithm DPO logicality faithfulness completeness human-valued dimensions