title: SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in
    Language Model Preference Learning
summary: Aligning language models with human preferences relies on pairwise preference
  datasets. While some studies suggest that on-policy data consistently
  outperforms off -policy data for preference learning, others indicate that the
  advantages of on-policy data may be task-dependent, highlighting the need for a
  systematic exploration of their interplay.
    In this work, we show that on-policy and off-policy data offer complementary
  strengths in preference optimization: on-policy data is particularly effective
  for reasoning tasks like math and coding, while off-policy data performs better
  on open-ended tasks such as creative writing and making personal
  recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach
  to combine the complementary strengths of on-policy and off-policy preference
  learning by simply mixing these two data sources. Our empirical results across
  diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves
  language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO
  and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it
  outperforms prior approaches that are much more complex in combining on- and
  off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.
paper_authors._id: 681a1efda513c2226dff667c
paper_authors.name: Tianjian Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T12:57:55.901Z
paper_authors._id: 62bc72d464b7a623f3a242e9
paper_authors.avatarUrl: /avatars/db79848dc78a5c4927e27de4426c3b47.svg
paper_authors.fullname: Tianjian Li
paper_authors.type: user
paper_authors.user: dogtooth
paper_authors._id: 681a1efda513c2226dff667d
paper_authors.name: Daniel Khashabi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-14T12:58:03.184Z
paper_authors._id: 5f6540c65e78cc6b0ed3199d
paper_authors.avatarUrl: /avatars/0280d4df417855965a0964d22766c012.svg
paper_authors.fullname: Daniel Khashabi
paper_authors.type: user
paper_authors.user: danyaljj
paper_title: SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in
    Language Model Preference Learning
paper_summary: Aligning language models with human preferences relies on pairwise preference
  datasets. While some studies suggest that on-policy data consistently
  outperforms off -policy data for preference learning, others indicate that the
  advantages of on-policy data may be task-dependent, highlighting the need for a
  systematic exploration of their interplay.
    In this work, we show that on-policy and off-policy data offer complementary
  strengths in preference optimization: on-policy data is particularly effective
  for reasoning tasks like math and coding, while off-policy data performs better
  on open-ended tasks such as creative writing and making personal
  recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach
  to combine the complementary strengths of on-policy and off-policy preference
  learning by simply mixing these two data sources. Our empirical results across
  diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves
  language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO
  and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it
  outperforms prior approaches that are much more complex in combining on- and
  off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.
paper_ai_summary: A combination of on-policy and off-policy data enhances language model alignment across various tasks, outperforming complex integration methods.
paper_ai_keywords: pairwise preference datasets preference learning on-policy data off-policy data preference optimization SIMPLEMIX reasoning tasks open-ended tasks language model alignment DPO Alpaca Eval 2.0 HyPO DPO-Mix-P