title: Modulated Intervention Preference Optimization (MIPO): Keep the Easy,
    Refine the Difficult
summary: Preference optimization methods typically begin training with a well-trained
  SFT model as a reference model. In RLHF and DPO, a regularization term is used
  during the preference optimization process to prevent the policy model from
  deviating too far from the reference model's distribution, thereby avoiding the
  generation of anomalous responses. When the reference model is already
  well-aligned with the given data or only requires slight adjustments, this
  approach can produce a well-aligned model. However, if the reference model is
  not aligned with the given data and requires significant deviation from its
  current state, a regularization term may actually hinder the model alignment.
  In this study, we propose Modulated Intervention Preference
  Optimization (MIPO) to address this issue. MIPO modulates the degree of
  intervention from the reference model based on how well the given data is
  aligned with it. If the data is well-aligned, the intervention is increased to
  prevent the policy model from diverging significantly from reference model.
  Conversely, if the alignment is poor, the interference is reduced to facilitate
  more extensive training. We compare the performance of MIPO and DPO using
  Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental
  results demonstrate that MIPO consistently outperforms DPO across various
  evaluation scenarios.
paper_authors._id: 66fa2848f50b1d60294265e8
paper_authors.name: Cheolhun Jang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:06:09.218Z
paper_authors._id: 64757d6322b4dec4524c924c
paper_authors.avatarUrl: /avatars/51c1a93f44375e2604d4c23ca66600ad.svg
paper_authors.fullname: Cheolhun Jang
paper_authors.type: user
paper_authors.user: fetong
paper_title: Modulated Intervention Preference Optimization (MIPO): Keep the Easy,
    Refine the Difficult
paper_summary: Preference optimization methods typically begin training with a well-trained
  SFT model as a reference model. In RLHF and DPO, a regularization term is used
  during the preference optimization process to prevent the policy model from
  deviating too far from the reference model's distribution, thereby avoiding the
  generation of anomalous responses. When the reference model is already
  well-aligned with the given data or only requires slight adjustments, this
  approach can produce a well-aligned model. However, if the reference model is
  not aligned with the given data and requires significant deviation from its
  current state, a regularization term may actually hinder the model alignment.
  In this study, we propose Modulated Intervention Preference
  Optimization (MIPO) to address this issue. MIPO modulates the degree of
  intervention from the reference model based on how well the given data is
  aligned with it. If the data is well-aligned, the intervention is increased to
  prevent the policy model from diverging significantly from reference model.
  Conversely, if the alignment is poor, the interference is reduced to facilitate
  more extensive training. We compare the performance of MIPO and DPO using
  Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench. The experimental
  results demonstrate that MIPO consistently outperforms DPO across various
  evaluation scenarios.
paper_ai_keywords: Preference optimization SFT model RLHF DPO viewpoint policy model regularization term modulated intervention Modulated Intervention Preference Optimization (MIPO) Alpaca Eval 2.0 MT-Bench