Differential Information: An Information-Theoretic Perspective on
  Preference Optimization
Direct Preference Optimization (DPO) has become a standard technique for
aligning language models with human preferences in a supervised manner. Despite
its empirical success, the theoretical justification behind its log-ratio
reward parameterization remains incomplete. In this work, we address this gap
by utilizing the Differential Information Distribution (DID): a distribution
over token sequences that captures the information gained during policy
updates. First, we show that when preference labels encode the differential
information required to transform a reference policy into a target policy, the
log-ratio reward in DPO emerges as the uniquely optimal form for learning the
target policy via preference optimization. This result naturally yields a
closed-form expression for the optimal sampling distribution over rejected
responses. Second, we find that the condition for preferences to encode
differential information is fundamentally linked to an implicit assumption
regarding log-margin ordered policies-an inductive bias widely used in
preference optimization yet previously unrecognized. Finally, by analyzing the
entropy of the DID, we characterize how learning low-entropy differential
information reinforces the policy distribution, while high-entropy differential
information induces a smoothing effect, which explains the log-likelihood
displacement phenomenon. We validate our theoretical findings in synthetic
experiments and extend them to real-world instruction-following datasets. Our
results suggest that learning high-entropy differential information is crucial
for general instruction-following, while learning low-entropy differential
information benefits knowledge-intensive question answering. Overall, our work
presents a unifying perspective on the DPO objective, the structure of
preference data, and resulting policy behaviors through the lens of
differential information.
6839107ed762b7c617b0731b
Yunjae Won
claimed_verified
2025-05-30T06:54:04.071Z
67578bf874cf42cdedbf00df
/avatars/5981300ee0025c876973f94c81a85b19.svg
Yunjae Won
user
yunjae-won
6839107ed762b7c617b0731c
Hyunji Lee
6839107ed762b7c617b0731d
Hyeonbin Hwang
6839107ed762b7c617b0731e
Minjoon Seo
Differential Information: An Information-Theoretic Perspective on
  Preference Optimization
Direct Preference Optimization (DPO) has become a standard technique for
aligning language models with human preferences in a supervised manner. Despite
its empirical success, the theoretical justification behind its log-ratio
reward parameterization remains incomplete. In this work, we address this gap
by utilizing the Differential Information Distribution (DID): a distribution
over token sequences that captures the information gained during policy
updates. First, we show that when preference labels encode the differential
information required to transform a reference policy into a target policy, the
log-ratio reward in DPO emerges as the uniquely optimal form for learning the
target policy via preference optimization. This result naturally yields a
closed-form expression for the optimal sampling distribution over rejected
responses. Second, we find that the condition for preferences to encode
differential information is fundamentally linked to an implicit assumption
regarding log-margin ordered policies-an inductive bias widely used in
preference optimization yet previously unrecognized. Finally, by analyzing the
entropy of the DID, we characterize how learning low-entropy differential
information reinforces the policy distribution, while high-entropy differential
information induces a smoothing effect, which explains the log-likelihood
displacement phenomenon. We validate our theoretical findings in synthetic
experiments and extend them to real-world instruction-following datasets. Our
results suggest that learning high-entropy differential information is crucial
for general instruction-following, while learning low-entropy differential
information benefits knowledge-intensive question answering. Overall, our work
presents a unifying perspective on the DPO objective, the structure of
preference data, and resulting policy behaviors through the lens of
differential information.
Theoretical analysis of Direct Preference Optimization (DPO) reveals that log-ratio reward parameterization is optimal for learning target policy via preference optimization, linked to log-margin ordered policies, and explains policy reinforcement and smoothing based on differential information entropy.
Direct Preference Optimization DPO log-ratio reward parameterization Differential Information Distribution DID token sequences policy updates preference labels target policy preference optimization log-margin ordered policies entropy differential information entropy log-likelihood displacement instruction-following datasets knowledge-intensive question answering