title: LongDPO: Unlock Better Long-form Generation Abilities for LLMs via
    Critique-augmented Stepwise Information
summary: Long-form generation is crucial for academic writing papers and repo-level
  code generation. Despite this, current models, including GPT-4o, still exhibit
  unsatisfactory performance. Existing methods that utilize preference learning
  with outcome supervision often fail to provide detailed feedback for extended
  contexts. This shortcoming can lead to content that does not fully satisfy
  query requirements, resulting in issues like length deviations, and diminished
  quality. In this paper, we propose enhancing long-form generation by
  incorporating process supervision. We employ Monte Carlo Tree Search to gather
  stepwise preference pairs, utilizing a global memory pool to maintain
  consistency. To address the issue of suboptimal candidate selection, we
  integrate external critiques to refine and improve the quality of the
  preference pairs. Finally, we apply step-level DPO using the collected stepwise
  preference pairs. Experimental results show that our method improves length and
  quality on long-form generation benchmarks, with almost lossless performance on
  general benchmarks across various model backbones.
paper_authors._id: 67a2c810ca39d45e49b9a07d
paper_authors.name: Bowen Ping
paper_authors._id: 67a2c810ca39d45e49b9a07e
paper_authors.name: Jiali Zeng
paper_authors._id: 67a2c810ca39d45e49b9a07f
paper_authors.name: Fandong Meng
paper_authors._id: 67a2c810ca39d45e49b9a080
paper_authors.name: Shuo Wang
paper_authors._id: 67a2c810ca39d45e49b9a081
paper_authors.name: Jie Zhou
paper_authors._id: 67a2c810ca39d45e49b9a082
paper_authors.name: Shanghang Zhang
paper_title: LongDPO: Unlock Better Long-form Generation Abilities for LLMs via
    Critique-augmented Stepwise Information
paper_summary: Long-form generation is crucial for academic writing papers and repo-level
  code generation. Despite this, current models, including GPT-4o, still exhibit
  unsatisfactory performance. Existing methods that utilize preference learning
  with outcome supervision often fail to provide detailed feedback for extended
  contexts. This shortcoming can lead to content that does not fully satisfy
  query requirements, resulting in issues like length deviations, and diminished
  quality. In this paper, we propose enhancing long-form generation by
  incorporating process supervision. We employ Monte Carlo Tree Search to gather
  stepwise preference pairs, utilizing a global memory pool to maintain
  consistency. To address the issue of suboptimal candidate selection, we
  integrate external critiques to refine and improve the quality of the
  preference pairs. Finally, we apply step-level DPO using the collected stepwise
  preference pairs. Experimental results show that our method improves length and
  quality on long-form generation benchmarks, with almost lossless performance on
  general benchmarks across various model backbones.
paper_ai_keywords: Monte Carlo Tree Search global memory pool external critiques step-level DPO preference pairs