DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich
  Paradigm for Direct Preference Optimization
The rapid rise of large language models (LLMs) has unlocked many applications
but also underscores the challenge of aligning them with diverse values and
preferences. Direct Preference Optimization (DPO) is central to alignment but
constrained by fixed divergences and limited feature transformations. We
propose DPO-Kernels, which integrates kernel methods to address these issues
through four key contributions: (i) Kernelized Representations with polynomial,
RBF, Mahalanobis, and spectral kernels for richer transformations, plus a
hybrid loss combining embedding-based and probability-based objectives; (ii)
Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya,
Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven
Selection metrics that automatically choose the best kernel-divergence pair;
and (iv) a Hierarchical Mixture of Kernels for both local precision and global
modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance
in factuality, safety, reasoning, and instruction following. Grounded in
Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization
for LLMs, offering a comprehensive resource for further alignment research.
677f70b49fd1b42991dbf051
Amitava Das
admin_assigned
2025-01-09T20:32:21.610Z
637537dc08eebfdd0a36e561
/avatars/70f800e80bac457649b1a4deaf6291ba.svg
Amitava Das
user
dramitavadas
677f70b49fd1b42991dbf052
Suranjana Trivedy
677f70b49fd1b42991dbf053
Danush Khanna
admin_assigned
2025-01-09T20:32:34.248Z
64731b1371f07ae738cffed6
https://cdn-avatars.huggingface.co/v1/production/uploads/64731b1371f07ae738cffed6/BoEiEgNrNZFXZPG5guWsu.jpeg
Danush Khanna
user
danushkhanna
677f70b49fd1b42991dbf054
Rajarshi Roy
claimed_verified
2025-01-14T08:30:09.995Z
64d752bb050438e3b96ce835
https://cdn-avatars.huggingface.co/v1/production/uploads/64d752bb050438e3b96ce835/Fs_kY1Fb8UwYsS42LhFCJ.jpeg
Rajarshi Roy
user
Rajarshi-Roy-research
677f70b49fd1b42991dbf055
Gurpreet Singh
677f70b49fd1b42991dbf056
Basab Ghosh
claimed_verified
2025-01-09T19:57:48.729Z
638f1a6cc4444c6ca86fc4b4
https://cdn-avatars.huggingface.co/v1/production/uploads/638f1a6cc4444c6ca86fc4b4/t_2zcBUdcyNvS9KlcybPx.jpeg
Basab Ghosh
user
basab1142
677f70b49fd1b42991dbf057
Yaswanth Narsupalli
claimed_verified
2025-01-10T08:21:29.527Z
64f6dc785f2dee8a6b9fcada
/avatars/7e9930c141eebdc9fcaddf2fd1d3a53c.svg
Narsupalli Yaswanth
user
yaswanth-iitkgp
677f70b49fd1b42991dbf058
Vinija Jain
admin_assigned
2025-01-09T20:33:06.982Z
6517bfecce732bf33a29d04b
/avatars/b6534a540fa10199df8f9acc497083d5.svg
Vinija Jain
user
Vinija
677f70b49fd1b42991dbf059
Vasu Sharma
admin_assigned
2025-01-09T20:33:14.847Z
65492730a3d1682f79f3ab7a
/avatars/fc62605bdf10947dab393305c51ace96.svg
Vasu Sharma
user
vasusharma55
677f70b49fd1b42991dbf05a
Aishwarya Naresh Reganti
677f70b49fd1b42991dbf05b
Aman Chadha
claimed_verified
2025-01-09T10:06:06.459Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich
  Paradigm for Direct Preference Optimization
The rapid rise of large language models (LLMs) has unlocked many applications
but also underscores the challenge of aligning them with diverse values and
preferences. Direct Preference Optimization (DPO) is central to alignment but
constrained by fixed divergences and limited feature transformations. We
propose DPO-Kernels, which integrates kernel methods to address these issues
through four key contributions: (i) Kernelized Representations with polynomial,
RBF, Mahalanobis, and spectral kernels for richer transformations, plus a
hybrid loss combining embedding-based and probability-based objectives; (ii)
Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya,
Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven
Selection metrics that automatically choose the best kernel-divergence pair;
and (iv) a Hierarchical Mixture of Kernels for both local precision and global
modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance
in factuality, safety, reasoning, and instruction following. Grounded in
Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization
for LLMs, offering a comprehensive resource for further alignment research.
DPO-Kernels integrates kernel methods into Direct Preference Optimization to enhance LLM alignment through richer representations, alternative divergences, automated selection, and hierarchical kernels.
LLMs Direct Preference Optimization DPO-Kernels Kernelized Representations polynomial kernels RBF kernels Mahalanobis kernels spectral kernels hybrid loss embedding-based objectives probability-based objectives Divergence Alternatives Jensen-Shannon divergence Hellinger divergence Renyi divergence Bhattacharyya divergence Wasserstein divergence f-divergences Data-Driven Selection metrics Hierarchical Mixture of Kernels Heavy-Tailed Self-Regularization