DRAGON: Distributional Rewards Optimize Diffusion Generative Models
We present Distributional RewArds for Generative OptimizatioN (DRAGON), a
versatile framework for fine-tuning media generation models towards a desired
outcome. Compared with traditional reinforcement learning with human feedback
(RLHF) or pairwise preference approaches such as direct preference optimization
(DPO), DRAGON is more flexible. It can optimize reward functions that evaluate
either individual examples or distributions of them, making it compatible with
a broad spectrum of instance-wise, instance-to-distribution, and
distribution-to-distribution rewards. Leveraging this versatility, we construct
novel reward functions by selecting an encoder and a set of reference examples
to create an exemplar distribution. When cross-modality encoders such as CLAP
are used, the reference examples may be of a different modality (e.g., text
versus audio). Then, DRAGON gathers online and on-policy generations, scores
them to construct a positive demonstration set and a negative set, and
leverages the contrast between the two sets to maximize the reward. For
evaluation, we fine-tune an audio-domain text-to-music diffusion model with 20
different reward functions, including a custom music aesthetics model, CLAP
score, Vendi diversity, and Frechet audio distance (FAD). We further compare
instance-wise (per-song) and full-dataset FAD settings while ablating multiple
FAD encoders and reference sets. Over all 20 target rewards, DRAGON achieves an
81.45% average win rate. Moreover, reward functions based on exemplar sets
indeed enhance generations and are comparable to model-based rewards. With an
appropriate exemplar set, DRAGON achieves a 60.95% human-voted music quality
win rate without training on human preference annotations. As such, DRAGON
exhibits a new approach to designing and optimizing reward functions for
improving human-perceived quality. Sound examples at
https://ml-dragon.github.io/web.
68070e1895e06498588c4497
Yatong Bai
claimed_verified
2025-04-22T09:50:50.514Z
63239bd492e07e3ca2068a16
https://cdn-avatars.huggingface.co/v1/production/uploads/63239bd492e07e3ca2068a16/OFESFFWQVHC1EtJpnGOWv.jpeg
Yatong Bai
user
Bai-YT
68070e1895e06498588c4498
Jonah Casebeer
extracted_confirmed
2025-04-23T17:53:28.353Z
67575907a1d686e1662863b3
/avatars/d3480b64822b1fa68ce24659e8a5cc7f.svg
Jonah Casebeer
user
jmcasebeer
68070e1895e06498588c4499
Somayeh Sojoudi
extracted_confirmed
2025-04-25T07:35:25.582Z
680b3a1acf67477f2c126811
/avatars/334b8e15a464ec633c4ddedd654d25b1.svg
Somayeh Sojoudi
user
Sojoudi
68070e1895e06498588c449a
Nicholas J. Bryan
extracted_confirmed
2025-04-22T03:35:40.134Z
648119ee7a741c7f33f49f25
/avatars/183d2143ba20e1cc8712c63c055aadd7.svg
Nicholas J. Bryan
user
Njb
DRAGON: Distributional Rewards Optimize Diffusion Generative Models
We present Distributional RewArds for Generative OptimizatioN (DRAGON), a
versatile framework for fine-tuning media generation models towards a desired
outcome. Compared with traditional reinforcement learning with human feedback
(RLHF) or pairwise preference approaches such as direct preference optimization
(DPO), DRAGON is more flexible. It can optimize reward functions that evaluate
either individual examples or distributions of them, making it compatible with
a broad spectrum of instance-wise, instance-to-distribution, and
distribution-to-distribution rewards. Leveraging this versatility, we construct
novel reward functions by selecting an encoder and a set of reference examples
to create an exemplar distribution. When cross-modality encoders such as CLAP
are used, the reference examples may be of a different modality (e.g., text
versus audio). Then, DRAGON gathers online and on-policy generations, scores
them to construct a positive demonstration set and a negative set, and
leverages the contrast between the two sets to maximize the reward. For
evaluation, we fine-tune an audio-domain text-to-music diffusion model with 20
different reward functions, including a custom music aesthetics model, CLAP
score, Vendi diversity, and Frechet audio distance (FAD). We further compare
instance-wise (per-song) and full-dataset FAD settings while ablating multiple
FAD encoders and reference sets. Over all 20 target rewards, DRAGON achieves an
81.45% average win rate. Moreover, reward functions based on exemplar sets
indeed enhance generations and are comparable to model-based rewards. With an
appropriate exemplar set, DRAGON achieves a 60.95% human-voted music quality
win rate without training on human preference annotations. As such, DRAGON
exhibits a new approach to designing and optimizing reward functions for
improving human-perceived quality. Sound examples at
https://ml-dragon.github.io/web.
DRAGON is a flexible framework for fine-tuning generative models using distributional rewards, outperforming traditional methods in optimizing human-perceived quality without extensive preference annotations.
Distributional RewArds for Generative OptimizatioN DRAGON reinforcement learning with human feedback RLHF direct preference optimization DPO reward functions exemplar distribution cross-modality encoders CLAP online and on-policy generations demonstration set negative set contrast audio-domain text-to-music diffusion model music aesthetics model CLAP score Vendi diversity Frechet audio distance human-voted music quality