title: Differential Information: An Information-Theoretic Perspective on
    Preference Optimization
summary: Direct Preference Optimization (DPO) has become a standard technique for
  aligning language models with human preferences in a supervised manner. Despite
  its empirical success, the theoretical justification behind its log-ratio
  reward parameterization remains incomplete. In this work, we address this gap
  by utilizing the Differential Information Distribution (DID): a distribution
  over token sequences that captures the information gained during policy
  updates. First, we show that when preference labels encode the differential
  information required to transform a reference policy into a target policy, the
  log-ratio reward in DPO emerges as the uniquely optimal form for learning the
  target policy via preference optimization. This result naturally yields a
  closed-form expression for the optimal sampling distribution over rejected
  responses. Second, we find that the condition for preferences to encode
  differential information is fundamentally linked to an implicit assumption
  regarding log-margin ordered policies-an inductive bias widely used in
  preference optimization yet previously unrecognized. Finally, by analyzing the
  entropy of the DID, we characterize how learning low-entropy differential
  information reinforces the policy distribution, while high-entropy differential
  information induces a smoothing effect, which explains the log-likelihood
  displacement phenomenon. We validate our theoretical findings in synthetic
  experiments and extend them to real-world instruction-following datasets. Our
  results suggest that learning high-entropy differential information is crucial
  for general instruction-following, while learning low-entropy differential
  information benefits knowledge-intensive question answering. Overall, our work
  presents a unifying perspective on the DPO objective, the structure of
  preference data, and resulting policy behaviors through the lens of
  differential information.
paper_authors._id: 6839107ed762b7c617b0731b
paper_authors.name: Yunjae Won
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-30T06:54:04.071Z
paper_authors._id: 67578bf874cf42cdedbf00df
paper_authors.avatarUrl: /avatars/5981300ee0025c876973f94c81a85b19.svg
paper_authors.fullname: Yunjae Won
paper_authors.type: user
paper_authors.user: yunjae-won
paper_authors._id: 6839107ed762b7c617b0731c
paper_authors.name: Hyunji Lee
paper_authors._id: 6839107ed762b7c617b0731d
paper_authors.name: Hyeonbin Hwang
paper_authors._id: 6839107ed762b7c617b0731e
paper_authors.name: Minjoon Seo
paper_title: Differential Information: An Information-Theoretic Perspective on
    Preference Optimization
paper_summary: Direct Preference Optimization (DPO) has become a standard technique for
  aligning language models with human preferences in a supervised manner. Despite
  its empirical success, the theoretical justification behind its log-ratio
  reward parameterization remains incomplete. In this work, we address this gap
  by utilizing the Differential Information Distribution (DID): a distribution
  over token sequences that captures the information gained during policy
  updates. First, we show that when preference labels encode the differential
  information required to transform a reference policy into a target policy, the
  log-ratio reward in DPO emerges as the uniquely optimal form for learning the
  target policy via preference optimization. This result naturally yields a
  closed-form expression for the optimal sampling distribution over rejected
  responses. Second, we find that the condition for preferences to encode
  differential information is fundamentally linked to an implicit assumption
  regarding log-margin ordered policies-an inductive bias widely used in
  preference optimization yet previously unrecognized. Finally, by analyzing the
  entropy of the DID, we characterize how learning low-entropy differential
  information reinforces the policy distribution, while high-entropy differential
  information induces a smoothing effect, which explains the log-likelihood
  displacement phenomenon. We validate our theoretical findings in synthetic
  experiments and extend them to real-world instruction-following datasets. Our
  results suggest that learning high-entropy differential information is crucial
  for general instruction-following, while learning low-entropy differential
  information benefits knowledge-intensive question answering. Overall, our work
  presents a unifying perspective on the DPO objective, the structure of
  preference data, and resulting policy behaviors through the lens of
  differential information.
paper_ai_summary: Theoretical analysis of Direct Preference Optimization (DPO) reveals that log-ratio reward parameterization is optimal for learning target policy via preference optimization, linked to log-margin ordered policies, and explains policy reinforcement and smoothing based on differential information entropy.
paper_ai_keywords: Direct Preference Optimization DPO log-ratio reward parameterization Differential Information Distribution DID token sequences policy updates preference labels target policy preference optimization log-margin ordered policies entropy differential information entropy log-likelihood displacement instruction-following datasets knowledge-intensive question answering