LongReward: Improving Long-context Large Language Models with AI
  Feedback
Though significant advancements have been achieved in developing long-context
large language models (LLMs), the compromised quality of LLM-synthesized data
for supervised fine-tuning (SFT) often affects the long-context performance of
SFT models and leads to inherent limitations. In principle, reinforcement
learning (RL) with appropriate reward signals can further enhance models'
capacities. However, how to obtain reliable rewards in long-context scenarios
remains unexplored. To this end, we propose LongReward, a novel method that
utilizes an off-the-shelf LLM to provide rewards for long-context model
responses from four human-valued dimensions: helpfulness, logicality,
faithfulness, and completeness, each with a carefully designed assessment
pipeline. By combining LongReward and offline RL algorithm DPO, we are able to
effectively improve long-context SFT models. Our experiments indicate that
LongReward not only significantly improves models' long-context performance but
also enhances their ability to follow short instructions. We also find that
long-context DPO with LongReward and conventional short-context DPO can be used
together without hurting either one's performance.
67204419dc8f6146490a8253
Jiajie Zhang
67204419dc8f6146490a8254
Zhongni Hou
67204419dc8f6146490a8255
Xin Lv
67204419dc8f6146490a8256
Shulin Cao
67204419dc8f6146490a8257
Zhenyu Hou
claimed_verified
2025-01-23T09:22:00.783Z
62b196646d3d059f40c3df19
/avatars/dbddf54ae949437223f3a438d30ef653.svg
Zhenyu Hou
user
think2try
67204419dc8f6146490a8258
Yilin Niu
67204419dc8f6146490a8259
Lei Hou
67204419dc8f6146490a825a
Yuxiao Dong
67204419dc8f6146490a825b
Ling Feng
67204419dc8f6146490a825c
Juanzi Li
LongReward: Improving Long-context Large Language Models with AI
  Feedback
Though significant advancements have been achieved in developing long-context
large language models (LLMs), the compromised quality of LLM-synthesized data
for supervised fine-tuning (SFT) often affects the long-context performance of
SFT models and leads to inherent limitations. In principle, reinforcement
learning (RL) with appropriate reward signals can further enhance models'
capacities. However, how to obtain reliable rewards in long-context scenarios
remains unexplored. To this end, we propose LongReward, a novel method that
utilizes an off-the-shelf LLM to provide rewards for long-context model
responses from four human-valued dimensions: helpfulness, logicality,
faithfulness, and completeness, each with a carefully designed assessment
pipeline. By combining LongReward and offline RL algorithm DPO, we are able to
effectively improve long-context SFT models. Our experiments indicate that
LongReward not only significantly improves models' long-context performance but
also enhances their ability to follow short instructions. We also find that
long-context DPO with LongReward and conventional short-context DPO can be used
together without hurting either one's performance.
A novel reward-based method called LongReward enhances long-context performance of supervised fine-tuned models by leveraging human-valued criteria and an off-the-shelf LLM, and it can be combined with offline RL algorithms like DPO effectively.
long-context large language models supervised fine-tuning reinforcement learning reward signals LongReward assessment pipeline offline RL algorithm DPO logicality faithfulness completeness human-valued dimensions