title: Margin-aware Preference Optimization for Aligning Diffusion Models
    without Reference
summary: Modern alignment techniques based on human preferences, such as RLHF and DPO,
  typically employ divergence regularization relative to the reference model to
  ensure training stability. However, this often limits the flexibility of models
  during alignment, especially when there is a clear distributional discrepancy
  between the preference data and the reference model. In this paper, we focus on
  the alignment of recent text-to-image diffusion models, such as Stable
  Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a
  significant problem in aligning these models due to the unstructured nature of
  visual modalities: e.g., a preference for a particular stylistic aspect can
  easily induce such a discrepancy. Motivated by this observation, we propose a
  novel and memory-friendly preference alignment method for diffusion models that
  does not depend on any reference model, coined margin-aware preference
  optimization (MaPO). MaPO jointly maximizes the likelihood margin between the
  preferred and dispreferred image sets and the likelihood of the preferred sets,
  simultaneously learning general stylistic features and preferences. For
  evaluation, we introduce two new pairwise preference datasets, which comprise
  self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating
  diverse scenarios of reference mismatch. Our experiments validate that MaPO can
  significantly improve alignment on Pick-Style and Pick-Safety and general
  preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and
  other existing methods. Our code, models, and datasets are publicly available
  via https://mapo-t2i.github.io
paper_authors._id: 6667e515fd95ddf66cab8fe8
paper_authors.name: Jiwoo Hong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T13:26:34.509Z
paper_authors._id: 6415c043486c7c9a5d151583
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6415c043486c7c9a5d151583/fUdYFh6iVh57swCkBEy-y.jpeg
paper_authors.fullname: Jiwoo Hong
paper_authors.type: user
paper_authors.user: JW17
paper_authors._id: 6667e515fd95ddf66cab8fe9
paper_authors.name: Sayak Paul
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-11T09:21:16.755Z
paper_authors._id: 5f7fbd813e94f16a85448745
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
paper_authors.fullname: Sayak Paul
paper_authors.type: user
paper_authors.user: sayakpaul
paper_authors._id: 6667e515fd95ddf66cab8fea
paper_authors.name: Noah Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-20T07:48:42.960Z
paper_authors._id: 6296fe6f7d586f66c3957f33
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6296fe6f7d586f66c3957f33/lCHe1mBVRzQ1sKsXD2OUd.jpeg
paper_authors.fullname: Noah Lee
paper_authors.type: user
paper_authors.user: nlee-208
paper_authors._id: 6667e515fd95ddf66cab8feb
paper_authors.name: Kashif Rasul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-11T15:13:41.779Z
paper_authors._id: 629f3b18ee05727ce328ccbe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669189789447-629f3b18ee05727ce328ccbe.jpeg
paper_authors.fullname: Kashif Rasul
paper_authors.type: user
paper_authors.user: kashif
paper_authors._id: 6667e515fd95ddf66cab8fec
paper_authors.name: James Thorne
paper_authors._id: 6667e515fd95ddf66cab8fed
paper_authors.name: Jongheon Jeong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-12T07:30:12.340Z
paper_authors._id: 660f22b4a8c77694a9c49486
paper_authors.avatarUrl: /avatars/718727912a7fdebb7daf696776746839.svg
paper_authors.fullname: Jongheon Jeong
paper_authors.type: user
paper_authors.user: jh-jeong
paper_title: Margin-aware Preference Optimization for Aligning Diffusion Models
    without Reference
paper_summary: Modern alignment techniques based on human preferences, such as RLHF and DPO,
  typically employ divergence regularization relative to the reference model to
  ensure training stability. However, this often limits the flexibility of models
  during alignment, especially when there is a clear distributional discrepancy
  between the preference data and the reference model. In this paper, we focus on
  the alignment of recent text-to-image diffusion models, such as Stable
  Diffusion XL (SDXL), and find that this "reference mismatch" is indeed a
  significant problem in aligning these models due to the unstructured nature of
  visual modalities: e.g., a preference for a particular stylistic aspect can
  easily induce such a discrepancy. Motivated by this observation, we propose a
  novel and memory-friendly preference alignment method for diffusion models that
  does not depend on any reference model, coined margin-aware preference
  optimization (MaPO). MaPO jointly maximizes the likelihood margin between the
  preferred and dispreferred image sets and the likelihood of the preferred sets,
  simultaneously learning general stylistic features and preferences. For
  evaluation, we introduce two new pairwise preference datasets, which comprise
  self-generated image pairs from SDXL, Pick-Style and Pick-Safety, simulating
  diverse scenarios of reference mismatch. Our experiments validate that MaPO can
  significantly improve alignment on Pick-Style and Pick-Safety and general
  preference alignment when used with Pick-a-Pic v2, surpassing the base SDXL and
  other existing methods. Our code, models, and datasets are publicly available
  via https://mapo-t2i.github.io
paper_ai_keywords: RLHF DPO divergence regularization reference model training stability text-to-image diffusion models Stable Diffusion XL (SDXL) reference mismatch visual modalities stylistic aspect distributional discrepancy preference alignment margin-aware preference optimization (MaPO) likelihood margin self-generated image pairs pairwise preference datasets Pick-Style Pick-Safety Pick-a-Pic v2