title: DRAGON: Distributional Rewards Optimize Diffusion Generative Models
summary: We present Distributional RewArds for Generative OptimizatioN (DRAGON), a
  versatile framework for fine-tuning media generation models towards a desired
  outcome. Compared with traditional reinforcement learning with human feedback
  (RLHF) or pairwise preference approaches such as direct preference optimization
  (DPO), DRAGON is more flexible. It can optimize reward functions that evaluate
  either individual examples or distributions of them, making it compatible with
  a broad spectrum of instance-wise, instance-to-distribution, and
  distribution-to-distribution rewards. Leveraging this versatility, we construct
  novel reward functions by selecting an encoder and a set of reference examples
  to create an exemplar distribution. When cross-modality encoders such as CLAP
  are used, the reference examples may be of a different modality (e.g., text
  versus audio). Then, DRAGON gathers online and on-policy generations, scores
  them to construct a positive demonstration set and a negative set, and
  leverages the contrast between the two sets to maximize the reward. For
  evaluation, we fine-tune an audio-domain text-to-music diffusion model with 20
  different reward functions, including a custom music aesthetics model, CLAP
  score, Vendi diversity, and Frechet audio distance (FAD). We further compare
  instance-wise (per-song) and full-dataset FAD settings while ablating multiple
  FAD encoders and reference sets. Over all 20 target rewards, DRAGON achieves an
  81.45% average win rate. Moreover, reward functions based on exemplar sets
  indeed enhance generations and are comparable to model-based rewards. With an
  appropriate exemplar set, DRAGON achieves a 60.95% human-voted music quality
  win rate without training on human preference annotations. As such, DRAGON
  exhibits a new approach to designing and optimizing reward functions for
  improving human-perceived quality. Sound examples at
  https://ml-dragon.github.io/web.
paper_authors._id: 68070e1895e06498588c4497
paper_authors.name: Yatong Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-22T09:50:50.514Z
paper_authors._id: 63239bd492e07e3ca2068a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63239bd492e07e3ca2068a16/OFESFFWQVHC1EtJpnGOWv.jpeg
paper_authors.fullname: Yatong Bai
paper_authors.type: user
paper_authors.user: Bai-YT
paper_authors._id: 68070e1895e06498588c4498
paper_authors.name: Jonah Casebeer
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-23T17:53:28.353Z
paper_authors._id: 67575907a1d686e1662863b3
paper_authors.avatarUrl: /avatars/d3480b64822b1fa68ce24659e8a5cc7f.svg
paper_authors.fullname: Jonah Casebeer
paper_authors.type: user
paper_authors.user: jmcasebeer
paper_authors._id: 68070e1895e06498588c4499
paper_authors.name: Somayeh Sojoudi
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-25T07:35:25.582Z
paper_authors._id: 680b3a1acf67477f2c126811
paper_authors.avatarUrl: /avatars/334b8e15a464ec633c4ddedd654d25b1.svg
paper_authors.fullname: Somayeh Sojoudi
paper_authors.type: user
paper_authors.user: Sojoudi
paper_authors._id: 68070e1895e06498588c449a
paper_authors.name: Nicholas J. Bryan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-04-22T03:35:40.134Z
paper_authors._id: 648119ee7a741c7f33f49f25
paper_authors.avatarUrl: /avatars/183d2143ba20e1cc8712c63c055aadd7.svg
paper_authors.fullname: Nicholas J. Bryan
paper_authors.type: user
paper_authors.user: Njb
paper_title: DRAGON: Distributional Rewards Optimize Diffusion Generative Models
paper_summary: We present Distributional RewArds for Generative OptimizatioN (DRAGON), a
  versatile framework for fine-tuning media generation models towards a desired
  outcome. Compared with traditional reinforcement learning with human feedback
  (RLHF) or pairwise preference approaches such as direct preference optimization
  (DPO), DRAGON is more flexible. It can optimize reward functions that evaluate
  either individual examples or distributions of them, making it compatible with
  a broad spectrum of instance-wise, instance-to-distribution, and
  distribution-to-distribution rewards. Leveraging this versatility, we construct
  novel reward functions by selecting an encoder and a set of reference examples
  to create an exemplar distribution. When cross-modality encoders such as CLAP
  are used, the reference examples may be of a different modality (e.g., text
  versus audio). Then, DRAGON gathers online and on-policy generations, scores
  them to construct a positive demonstration set and a negative set, and
  leverages the contrast between the two sets to maximize the reward. For
  evaluation, we fine-tune an audio-domain text-to-music diffusion model with 20
  different reward functions, including a custom music aesthetics model, CLAP
  score, Vendi diversity, and Frechet audio distance (FAD). We further compare
  instance-wise (per-song) and full-dataset FAD settings while ablating multiple
  FAD encoders and reference sets. Over all 20 target rewards, DRAGON achieves an
  81.45% average win rate. Moreover, reward functions based on exemplar sets
  indeed enhance generations and are comparable to model-based rewards. With an
  appropriate exemplar set, DRAGON achieves a 60.95% human-voted music quality
  win rate without training on human preference annotations. As such, DRAGON
  exhibits a new approach to designing and optimizing reward functions for
  improving human-perceived quality. Sound examples at
  https://ml-dragon.github.io/web.
paper_ai_summary: DRAGON is a flexible framework for fine-tuning generative models using distributional rewards, outperforming traditional methods in optimizing human-perceived quality without extensive preference annotations.
paper_ai_keywords: Distributional RewArds for Generative OptimizatioN DRAGON reinforcement learning with human feedback RLHF direct preference optimization DPO reward functions exemplar distribution cross-modality encoders CLAP online and on-policy generations demonstration set negative set contrast audio-domain text-to-music diffusion model music aesthetics model CLAP score Vendi diversity Frechet audio distance human-voted music quality