title: Aligning Diffusion Models with Noise-Conditioned Perception
summary: Recent advancements in human preference optimization, initially developed for
  Language Models (LMs), have shown promise for text-to-image Diffusion Models,
  enhancing prompt alignment, visual appeal, and user preference. Unlike LMs,
  Diffusion Models typically optimize in pixel or VAE space, which does not align
  well with human perception, leading to slower and less efficient training
  during the preference alignment stage. We propose using a perceptual objective
  in the U-Net embedding space of the diffusion model to address these issues.
  Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct
  Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and
  supervised fine-tuning (SFT) within this embedding space. This method
  significantly outperforms standard latent-space implementations across various
  metrics, including quality and computational cost. For SDXL, our approach
  provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt
  following against original open-sourced SDXL-DPO on the PartiPrompts dataset,
  while significantly reducing compute. Our approach not only improves the
  efficiency and quality of human preference alignment for diffusion models but
  is also easily integrable with other optimization techniques. The training code
  and LoRA weights will be available here:
  https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
paper_authors._id: 667bb9fcb9e88cc05404e766
paper_authors.name: Alexander Gambashidze
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-27T19:04:13.562Z
paper_authors._id: 6192657ba9638054a9818f04
paper_authors.avatarUrl: /avatars/35f6f6eb2e8f6b283034632a141c2670.svg
paper_authors.fullname: Alexander Gambashidze
paper_authors.type: user
paper_authors.user: alexgambashidze
paper_authors._id: 667bb9fcb9e88cc05404e767
paper_authors.name: Anton Kulikov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-26T10:00:33.181Z
paper_authors._id: 6650c0bd87144bf8106682fb
paper_authors.avatarUrl: /avatars/6f241c967b045c6d18fba8efd0fee3c6.svg
paper_authors.fullname: Anton Kulikov
paper_authors.type: user
paper_authors.user: neolizee
paper_authors._id: 667bb9fcb9e88cc05404e768
paper_authors.name: Yuriy Sosnin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-27T09:03:14.183Z
paper_authors._id: 6576ea2e66330867a1fd3fc2
paper_authors.avatarUrl: /avatars/27553edc0693ca6559437a05f3b5fb21.svg
paper_authors.fullname: Yuriy Sosnin
paper_authors.type: user
paper_authors.user: yuasosnin
paper_authors._id: 667bb9fcb9e88cc05404e769
paper_authors.name: Ilya Makarov
paper_title: Aligning Diffusion Models with Noise-Conditioned Perception
paper_summary: Recent advancements in human preference optimization, initially developed for
  Language Models (LMs), have shown promise for text-to-image Diffusion Models,
  enhancing prompt alignment, visual appeal, and user preference. Unlike LMs,
  Diffusion Models typically optimize in pixel or VAE space, which does not align
  well with human perception, leading to slower and less efficient training
  during the preference alignment stage. We propose using a perceptual objective
  in the U-Net embedding space of the diffusion model to address these issues.
  Our approach involves fine-tuning Stable Diffusion 1.5 and XL using Direct
  Preference Optimization (DPO), Contrastive Preference Optimization (CPO), and
  supervised fine-tuning (SFT) within this embedding space. This method
  significantly outperforms standard latent-space implementations across various
  metrics, including quality and computational cost. For SDXL, our approach
  provides 60.8\% general preference, 62.2\% visual appeal, and 52.1\% prompt
  following against original open-sourced SDXL-DPO on the PartiPrompts dataset,
  while significantly reducing compute. Our approach not only improves the
  efficiency and quality of human preference alignment for diffusion models but
  is also easily integrable with other optimization techniques. The training code
  and LoRA weights will be available here:
  https://huggingface.co/alexgambashidze/SDXL\_NCP-DPO\_v0.1
paper_ai_keywords: Diffusion Models U-Net embedding space Stable Diffusion 1.5 Stable Diffusion XL (SDXL) Direct Preference Optimization (DPO) Contrastive Preference Optimization (CPO) Supervised fine-tuning (SFT) perceptual objective latent-space implementations PartiPrompts dataset LoRA weights