title: Finding the Sweet Spot: Preference Data Construction for Scaling
    Preference Optimization
summary: Iterative data generation and model retraining are widely used to align large
  language models (LLMs). It typically involves a policy model to generate
  on-policy responses and a reward model to guide training data selection. Direct
  Preference Optimization (DPO) further enhances this process by constructing
  preference pairs of chosen and rejected responses. In this work, we aim to
  scale up the number of on-policy samples via repeated random sampling to
  improve alignment performance. Conventional practice selects the sample with
  the highest reward as chosen and the lowest as rejected for DPO. However, our
  experiments reveal that this strategy leads to a decline in performance
  as the sample size increases. To address this, we investigate preference data
  construction through the lens of underlying normal distribution of sample
  rewards. We categorize the reward space into seven representative points and
  systematically explore all 21 (C_7^2) pairwise combinations. Through
  evaluations on four models using AlpacaEval 2, we find that selecting the
  rejected response at reward position mu - 2sigma rather than the minimum
  reward, is crucial for optimal performance. We finally introduce a scalable
  preference data construction strategy that consistently enhances model
  performance as the sample scale increases.
paper_authors._id: 67bf243823f222a2cc2858d0
paper_authors.name: Yao Xiao
paper_authors._id: 67bf243823f222a2cc2858d1
paper_authors.name: Hai Ye
paper_authors._id: 67bf243823f222a2cc2858d2
paper_authors.name: Linyao Chen
paper_authors._id: 67bf243823f222a2cc2858d3
paper_authors.name: Hwee Tou Ng
paper_authors._id: 67bf243823f222a2cc2858d4
paper_authors.name: Lidong Bing
paper_authors._id: 67bf243823f222a2cc2858d5
paper_authors.name: Xiaoli Li
paper_authors._id: 67bf243823f222a2cc2858d6
paper_authors.name: Roy Ka-wei Lee
paper_title: Finding the Sweet Spot: Preference Data Construction for Scaling
    Preference Optimization
paper_summary: Iterative data generation and model retraining are widely used to align large
  language models (LLMs). It typically involves a policy model to generate
  on-policy responses and a reward model to guide training data selection. Direct
  Preference Optimization (DPO) further enhances this process by constructing
  preference pairs of chosen and rejected responses. In this work, we aim to
  scale up the number of on-policy samples via repeated random sampling to
  improve alignment performance. Conventional practice selects the sample with
  the highest reward as chosen and the lowest as rejected for DPO. However, our
  experiments reveal that this strategy leads to a decline in performance
  as the sample size increases. To address this, we investigate preference data
  construction through the lens of underlying normal distribution of sample
  rewards. We categorize the reward space into seven representative points and
  systematically explore all 21 (C_7^2) pairwise combinations. Through
  evaluations on four models using AlpacaEval 2, we find that selecting the
  rejected response at reward position mu - 2sigma rather than the minimum
  reward, is crucial for optimal performance. We finally introduce a scalable
  preference data construction strategy that consistently enhances model
  performance as the sample scale increases.
paper_ai_keywords: large language models (LLMs) policy model reward model Direct Preference Optimization (DPO) on-policy samples repeated random sampling decline preference pairs underlying normal distribution reward space pairwise combinations AlpacaEval 2 scalable preference data construction strategy