title: When Preferences Diverge: Aligning Diffusion Models with Minority-Aware
    Adaptive DPO
summary: In recent years, the field of image generation has witnessed significant
  advancements, particularly in fine-tuning methods that align models with
  universal human preferences. This paper explores the critical role of
  preference data in the training process of diffusion models, particularly in
  the context of Diffusion-DPO and its subsequent adaptations. We investigate the
  complexities surrounding universal human preferences in image generation,
  highlighting the subjective nature of these preferences and the challenges
  posed by minority samples in preference datasets. Through pilot experiments, we
  demonstrate the existence of minority samples and their detrimental effects on
  model performance. We propose Adaptive-DPO -- a novel approach that
  incorporates a minority-instance-aware metric into the DPO objective. This
  metric, which includes intra-annotator confidence and inter-annotator
  stability, distinguishes between majority and minority samples. We introduce an
  Adaptive-DPO loss function which improves the DPO loss in two ways: enhancing
  the model's learning of majority labels while mitigating the negative impact of
  minority samples. Our experiments demonstrate that this method effectively
  handles both synthetic minority data and real-world preference data, paving the
  way for more effective training methodologies in image generation tasks.
paper_authors._id: 67e0bb1665e294ad989334ea
paper_authors.name: Lingfan Zhang
paper_authors._id: 67e0bb1665e294ad989334eb
paper_authors.name: Chen Liu
paper_authors._id: 67e0bb1665e294ad989334ec
paper_authors.name: Chengming Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:49:08.734Z
paper_authors._id: 652fab9d04a34a9282bf29d6
paper_authors.avatarUrl: /avatars/cd5967b37ebb1225e9ae1d46f196e2e2.svg
paper_authors.fullname: Chengming Xu
paper_authors.type: user
paper_authors.user: ChengmingX
paper_authors._id: 67e0bb1665e294ad989334ed
paper_authors.name: Kai Hu
paper_authors._id: 67e0bb1665e294ad989334ee
paper_authors.name: Donghao Luo
paper_authors._id: 67e0bb1665e294ad989334ef
paper_authors.name: Chengjie Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:48:42.939Z
paper_authors._id: 65729087379284b904a6d81d
paper_authors.avatarUrl: /avatars/0e7968b1d9221d523a54e3ac787e449d.svg
paper_authors.fullname: Chengjie Wang
paper_authors.type: user
paper_authors.user: chengjie-wang
paper_authors._id: 67e0bb1665e294ad989334f0
paper_authors.name: Yanwei Fu
paper_authors._id: 67e0bb1665e294ad989334f1
paper_authors.name: Yuan Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-24T12:48:12.962Z
paper_authors._id: 664618d83a7681520570b58a
paper_authors.avatarUrl: /avatars/c03e25ee9bbf3504cdf2959acf81e0fe.svg
paper_authors.fullname: Yuan Yao
paper_authors.type: user
paper_authors.user: yuanyao
paper_title: When Preferences Diverge: Aligning Diffusion Models with Minority-Aware
    Adaptive DPO
paper_summary: In recent years, the field of image generation has witnessed significant
  advancements, particularly in fine-tuning methods that align models with
  universal human preferences. This paper explores the critical role of
  preference data in the training process of diffusion models, particularly in
  the context of Diffusion-DPO and its subsequent adaptations. We investigate the
  complexities surrounding universal human preferences in image generation,
  highlighting the subjective nature of these preferences and the challenges
  posed by minority samples in preference datasets. Through pilot experiments, we
  demonstrate the existence of minority samples and their detrimental effects on
  model performance. We propose Adaptive-DPO -- a novel approach that
  incorporates a minority-instance-aware metric into the DPO objective. This
  metric, which includes intra-annotator confidence and inter-annotator
  stability, distinguishes between majority and minority samples. We introduce an
  Adaptive-DPO loss function which improves the DPO loss in two ways: enhancing
  the model's learning of majority labels while mitigating the negative impact of
  minority samples. Our experiments demonstrate that this method effectively
  handles both synthetic minority data and real-world preference data, paving the
  way for more effective training methodologies in image generation tasks.
paper_ai_summary: A novel approach, Adaptive-DPO, enhances the training of diffusion models by incorporating a minority-instance-aware metric to effectively manage both synthetic and real-world preference data.
paper_ai_keywords: diffusion models Diffusion-DPO universal human preferences preference data minority samples intra-annotator confidence inter-annotator stability Adaptive-DPO synthetic minority data