LongDPO: Unlock Better Long-form Generation Abilities for LLMs via
  Critique-augmented Stepwise Information
Long-form generation is crucial for academic writing papers and repo-level
code generation. Despite this, current models, including GPT-4o, still exhibit
unsatisfactory performance. Existing methods that utilize preference learning
with outcome supervision often fail to provide detailed feedback for extended
contexts. This shortcoming can lead to content that does not fully satisfy
query requirements, resulting in issues like length deviations, and diminished
quality. In this paper, we propose enhancing long-form generation by
incorporating process supervision. We employ Monte Carlo Tree Search to gather
stepwise preference pairs, utilizing a global memory pool to maintain
consistency. To address the issue of suboptimal candidate selection, we
integrate external critiques to refine and improve the quality of the
preference pairs. Finally, we apply step-level DPO using the collected stepwise
preference pairs. Experimental results show that our method improves length and
quality on long-form generation benchmarks, with almost lossless performance on
general benchmarks across various model backbones.
67a2c810ca39d45e49b9a07d
Bowen Ping
67a2c810ca39d45e49b9a07e
Jiali Zeng
67a2c810ca39d45e49b9a07f
Fandong Meng
67a2c810ca39d45e49b9a080
Shuo Wang
67a2c810ca39d45e49b9a081
Jie Zhou
67a2c810ca39d45e49b9a082
Shanghang Zhang
LongDPO: Unlock Better Long-form Generation Abilities for LLMs via
  Critique-augmented Stepwise Information
Long-form generation is crucial for academic writing papers and repo-level
code generation. Despite this, current models, including GPT-4o, still exhibit
unsatisfactory performance. Existing methods that utilize preference learning
with outcome supervision often fail to provide detailed feedback for extended
contexts. This shortcoming can lead to content that does not fully satisfy
query requirements, resulting in issues like length deviations, and diminished
quality. In this paper, we propose enhancing long-form generation by
incorporating process supervision. We employ Monte Carlo Tree Search to gather
stepwise preference pairs, utilizing a global memory pool to maintain
consistency. To address the issue of suboptimal candidate selection, we
integrate external critiques to refine and improve the quality of the
preference pairs. Finally, we apply step-level DPO using the collected stepwise
preference pairs. Experimental results show that our method improves length and
quality on long-form generation benchmarks, with almost lossless performance on
general benchmarks across various model backbones.
Incorporating process supervision with Monte Carlo Tree Search and step-level DPO improves long-form generation quality and length consistency.
Monte Carlo Tree Search global memory pool stepwise preference pairs external critiques step-level DPO long-form generation length deviation quality improvement