The Crucial Role of Samplers in Online Direct Preference Optimization
<|reference_start|>The Crucial Role of Samplers in Online Direct Preference Optimization: Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the $\textit{optimization}$ properties, particularly the impact of samplers on its convergence rates, remain underexplored. In this paper, we provide a rigorous analysis of DPO's $\textit{convergence rates}$ with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves $\textit{linear}$ convergence, while our proposed online sampler achieves $\textit{quadratic}$ convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and $\textit{logit mixing}$, demonstrating significant improvements over previous approaches. On Safe-RLHF dataset, our method exhibits a $4.5$% improvement over vanilla DPO and a $3.0$% improvement over on-policy DPO; on Iterative-Prompt, our approach outperforms vanilla DPO, on-policy DPO, and Hybrid GSHF by over $4.2$%. Our results not only offer insights into the theoretical standing of DPO but also pave the way for potential algorithm designs in the future.<|reference_end|>
@article{shi2024the,
    title={The Crucial Role of Samplers in Online Direct Preference Optimization},
    author={Ruizhe Shi, Runlong Zhou, Simon S. Du},
    journal={arXiv preprint arXiv:2409.19605},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.19605},
    primaryClass={cs.LG cs.CL}
}