title: DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich
    Paradigm for Direct Preference Optimization
summary: The rapid rise of large language models (LLMs) has unlocked many applications
  but also underscores the challenge of aligning them with diverse values and
  preferences. Direct Preference Optimization (DPO) is central to alignment but
  constrained by fixed divergences and limited feature transformations. We
  propose DPO-Kernels, which integrates kernel methods to address these issues
  through four key contributions: (i) Kernelized Representations with polynomial,
  RBF, Mahalanobis, and spectral kernels for richer transformations, plus a
  hybrid loss combining embedding-based and probability-based objectives; (ii)
  Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya,
  Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven
  Selection metrics that automatically choose the best kernel-divergence pair;
  and (iv) a Hierarchical Mixture of Kernels for both local precision and global
  modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance
  in factuality, safety, reasoning, and instruction following. Grounded in
  Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization
  for LLMs, offering a comprehensive resource for further alignment research.
paper_authors._id: 677f70b49fd1b42991dbf051
paper_authors.name: Amitava Das
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:32:21.610Z
paper_authors._id: 637537dc08eebfdd0a36e561
paper_authors.avatarUrl: /avatars/70f800e80bac457649b1a4deaf6291ba.svg
paper_authors.fullname: Amitava Das
paper_authors.type: user
paper_authors.user: dramitavadas
paper_authors._id: 677f70b49fd1b42991dbf052
paper_authors.name: Suranjana Trivedy
paper_authors._id: 677f70b49fd1b42991dbf053
paper_authors.name: Danush Khanna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:32:34.248Z
paper_authors._id: 64731b1371f07ae738cffed6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64731b1371f07ae738cffed6/BoEiEgNrNZFXZPG5guWsu.jpeg
paper_authors.fullname: Danush Khanna
paper_authors.type: user
paper_authors.user: danushkhanna
paper_authors._id: 677f70b49fd1b42991dbf054
paper_authors.name: Rajarshi Roy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:30:09.995Z
paper_authors._id: 64d752bb050438e3b96ce835
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d752bb050438e3b96ce835/Fs_kY1Fb8UwYsS42LhFCJ.jpeg
paper_authors.fullname: Rajarshi Roy
paper_authors.type: user
paper_authors.user: Rajarshi-Roy-research
paper_authors._id: 677f70b49fd1b42991dbf055
paper_authors.name: Gurpreet Singh
paper_authors._id: 677f70b49fd1b42991dbf056
paper_authors.name: Basab Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T19:57:48.729Z
paper_authors._id: 638f1a6cc4444c6ca86fc4b4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638f1a6cc4444c6ca86fc4b4/t_2zcBUdcyNvS9KlcybPx.jpeg
paper_authors.fullname: Basab Ghosh
paper_authors.type: user
paper_authors.user: basab1142
paper_authors._id: 677f70b49fd1b42991dbf057
paper_authors.name: Yaswanth Narsupalli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:21:29.527Z
paper_authors._id: 64f6dc785f2dee8a6b9fcada
paper_authors.avatarUrl: /avatars/7e9930c141eebdc9fcaddf2fd1d3a53c.svg
paper_authors.fullname: Narsupalli Yaswanth
paper_authors.type: user
paper_authors.user: yaswanth-iitkgp
paper_authors._id: 677f70b49fd1b42991dbf058
paper_authors.name: Vinija Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:33:06.982Z
paper_authors._id: 6517bfecce732bf33a29d04b
paper_authors.avatarUrl: /avatars/b6534a540fa10199df8f9acc497083d5.svg
paper_authors.fullname: Vinija Jain
paper_authors.type: user
paper_authors.user: Vinija
paper_authors._id: 677f70b49fd1b42991dbf059
paper_authors.name: Vasu Sharma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-09T20:33:14.847Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 677f70b49fd1b42991dbf05a
paper_authors.name: Aishwarya Naresh Reganti
paper_authors._id: 677f70b49fd1b42991dbf05b
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:06:06.459Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_title: DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich
    Paradigm for Direct Preference Optimization
paper_summary: The rapid rise of large language models (LLMs) has unlocked many applications
  but also underscores the challenge of aligning them with diverse values and
  preferences. Direct Preference Optimization (DPO) is central to alignment but
  constrained by fixed divergences and limited feature transformations. We
  propose DPO-Kernels, which integrates kernel methods to address these issues
  through four key contributions: (i) Kernelized Representations with polynomial,
  RBF, Mahalanobis, and spectral kernels for richer transformations, plus a
  hybrid loss combining embedding-based and probability-based objectives; (ii)
  Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya,
  Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven
  Selection metrics that automatically choose the best kernel-divergence pair;
  and (iv) a Hierarchical Mixture of Kernels for both local precision and global
  modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance
  in factuality, safety, reasoning, and instruction following. Grounded in
  Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization
  for LLMs, offering a comprehensive resource for further alignment research.
paper_ai_keywords: direct preference optimization (DPO) DPO-Kernels kernel methods kernelized representations polynomial kernels RBF kernels Mahalanobis kernels spectral kernels hybrid loss embedding-based objectives probability-based objectives divergence alternatives jensen-shannon divergence hellinger divergence renyi divergence bhattacharyya divergence wasserstein divergence f-divergences data-driven selection metrics hierarchical mixture of kernels heavy-tailed self-regularization robust generalization