Diversity-Rewarded CFG Distillation
Generative models are transforming creative domains such as music generation,
with inference-time strategies like Classifier-Free Guidance (CFG) playing a
crucial role. However, CFG doubles inference cost while limiting originality
and diversity across generated contents. In this paper, we introduce
diversity-rewarded CFG distillation, a novel finetuning procedure that distills
the strengths of CFG while addressing its limitations. Our approach optimises
two training objectives: (1) a distillation objective, encouraging the model
alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL
objective with a diversity reward, promoting the generation of diverse outputs
for a given prompt. By finetuning, we learn model weights with the ability to
generate high-quality and diverse outputs, without any inference overhead. This
also unlocks the potential of weight-based model merging strategies: by
interpolating between the weights of two models (the first focusing on quality,
the second on diversity), we can control the quality-diversity trade-off at
deployment time, and even further boost performance. We conduct extensive
experiments on the MusicLM (Agostinelli et al., 2023) text-to-music generative
model, where our approach surpasses CFG in terms of quality-diversity Pareto
optimality. According to human evaluators, our finetuned-then-merged model
generates samples with higher quality-diversity than the base model augmented
with CFG. Explore our generations at
https://google-research.github.io/seanet/musiclm/diverse_music/.
6707aba2356f027ca984e789
Geoffrey Cideron
6707aba2356f027ca984e78a
Andrea Agostinelli
admin_assigned
2024-10-10T11:12:01.996Z
600e79032b417b1d53669bdb
/avatars/a60d5ae82340722285474b3a5d38e234.svg
andrea agostinelli
user
aagostinelli86
6707aba2356f027ca984e78b
Johan Ferret
admin_assigned
2024-10-10T11:12:08.489Z
65afb7dbdd6bdfd73cd8e609
/avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
Johan Ferret
user
ferretj
6707aba2356f027ca984e78c
Sertan Girgin
6707aba2356f027ca984e78d
Romuald Elie
6707aba2356f027ca984e78e
Olivier Bachem
admin_assigned
2024-10-10T11:12:20.214Z
667aac871bffb68706b4f62c
/avatars/2e2f2431f03a368f604c992d0d7ca57a.svg
Olivier Bachem
user
bachem
6707aba2356f027ca984e78f
Sarah Perrin
admin_assigned
2024-10-10T11:12:27.578Z
66328157b270ae503e91339b
/avatars/ea7a52060f5360f523ca28e137e85e33.svg
Sarah Perrin
user
Sper42
6707aba2356f027ca984e790
Alexandre Ramé
claimed_verified
2024-10-10T10:33:00.042Z
63c94ede00104ea998de19a6
/avatars/273959d87f0c67747588cf0700d64039.svg
Alexandre Rame
user
alexrame
Diversity-Rewarded CFG Distillation
Generative models are transforming creative domains such as music generation,
with inference-time strategies like Classifier-Free Guidance (CFG) playing a
crucial role. However, CFG doubles inference cost while limiting originality
and diversity across generated contents. In this paper, we introduce
diversity-rewarded CFG distillation, a novel finetuning procedure that distills
the strengths of CFG while addressing its limitations. Our approach optimises
two training objectives: (1) a distillation objective, encouraging the model
alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL
objective with a diversity reward, promoting the generation of diverse outputs
for a given prompt. By finetuning, we learn model weights with the ability to
generate high-quality and diverse outputs, without any inference overhead. This
also unlocks the potential of weight-based model merging strategies: by
interpolating between the weights of two models (the first focusing on quality,
the second on diversity), we can control the quality-diversity trade-off at
deployment time, and even further boost performance. We conduct extensive
experiments on the MusicLM (Agostinelli et al., 2023) text-to-music generative
model, where our approach surpasses CFG in terms of quality-diversity Pareto
optimality. According to human evaluators, our finetuned-then-merged model
generates samples with higher quality-diversity than the base model augmented
with CFG. Explore our generations at
https://google-research.github.io/seanet/musiclm/diverse_music/.
A new finetuning procedure called diversity-rewarded CFG distillation enhances generative models by balancing quality and diversity without inference overhead.
Classifier-Free Guidance (CFG) distillation objective RL objective diversity reward model merging quality-diversity Pareto optimality MusicLM