title: Diversity-Rewarded CFG Distillation
summary: Generative models are transforming creative domains such as music generation,
  with inference-time strategies like Classifier-Free Guidance (CFG) playing a
  crucial role. However, CFG doubles inference cost while limiting originality
  and diversity across generated contents. In this paper, we introduce
  diversity-rewarded CFG distillation, a novel finetuning procedure that distills
  the strengths of CFG while addressing its limitations. Our approach optimises
  two training objectives: (1) a distillation objective, encouraging the model
  alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL
  objective with a diversity reward, promoting the generation of diverse outputs
  for a given prompt. By finetuning, we learn model weights with the ability to
  generate high-quality and diverse outputs, without any inference overhead. This
  also unlocks the potential of weight-based model merging strategies: by
  interpolating between the weights of two models (the first focusing on quality,
  the second on diversity), we can control the quality-diversity trade-off at
  deployment time, and even further boost performance. We conduct extensive
  experiments on the MusicLM (Agostinelli et al., 2023) text-to-music generative
  model, where our approach surpasses CFG in terms of quality-diversity Pareto
  optimality. According to human evaluators, our finetuned-then-merged model
  generates samples with higher quality-diversity than the base model augmented
  with CFG. Explore our generations at
  https://google-research.github.io/seanet/musiclm/diverse_music/.
paper_authors._id: 6707aba2356f027ca984e789
paper_authors.name: Geoffrey Cideron
paper_authors._id: 6707aba2356f027ca984e78a
paper_authors.name: Andrea Agostinelli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:12:01.996Z
paper_authors._id: 600e79032b417b1d53669bdb
paper_authors.avatarUrl: /avatars/a60d5ae82340722285474b3a5d38e234.svg
paper_authors.fullname: andrea agostinelli
paper_authors.type: user
paper_authors.user: aagostinelli86
paper_authors._id: 6707aba2356f027ca984e78b
paper_authors.name: Johan Ferret
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:12:08.489Z
paper_authors._id: 65afb7dbdd6bdfd73cd8e609
paper_authors.avatarUrl: /avatars/b21069bc2d7ee4cc1508008e3c8ade64.svg
paper_authors.fullname: Johan Ferret
paper_authors.type: user
paper_authors.user: ferretj
paper_authors._id: 6707aba2356f027ca984e78c
paper_authors.name: Sertan Girgin
paper_authors._id: 6707aba2356f027ca984e78d
paper_authors.name: Romuald Elie
paper_authors._id: 6707aba2356f027ca984e78e
paper_authors.name: Olivier Bachem
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:12:20.214Z
paper_authors._id: 667aac871bffb68706b4f62c
paper_authors.avatarUrl: /avatars/2e2f2431f03a368f604c992d0d7ca57a.svg
paper_authors.fullname: Olivier Bachem
paper_authors.type: user
paper_authors.user: bachem
paper_authors._id: 6707aba2356f027ca984e78f
paper_authors.name: Sarah Perrin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T11:12:27.578Z
paper_authors._id: 66328157b270ae503e91339b
paper_authors.avatarUrl: /avatars/ea7a52060f5360f523ca28e137e85e33.svg
paper_authors.fullname: Sarah Perrin
paper_authors.type: user
paper_authors.user: Sper42
paper_authors._id: 6707aba2356f027ca984e790
paper_authors.name: Alexandre Ramé
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T10:33:00.042Z
paper_authors._id: 63c94ede00104ea998de19a6
paper_authors.avatarUrl: /avatars/273959d87f0c67747588cf0700d64039.svg
paper_authors.fullname: Alexandre Rame
paper_authors.type: user
paper_authors.user: alexrame
paper_title: Diversity-Rewarded CFG Distillation
paper_summary: Generative models are transforming creative domains such as music generation,
  with inference-time strategies like Classifier-Free Guidance (CFG) playing a
  crucial role. However, CFG doubles inference cost while limiting originality
  and diversity across generated contents. In this paper, we introduce
  diversity-rewarded CFG distillation, a novel finetuning procedure that distills
  the strengths of CFG while addressing its limitations. Our approach optimises
  two training objectives: (1) a distillation objective, encouraging the model
  alone (without CFG) to imitate the CFG-augmented predictions, and (2) an RL
  objective with a diversity reward, promoting the generation of diverse outputs
  for a given prompt. By finetuning, we learn model weights with the ability to
  generate high-quality and diverse outputs, without any inference overhead. This
  also unlocks the potential of weight-based model merging strategies: by
  interpolating between the weights of two models (the first focusing on quality,
  the second on diversity), we can control the quality-diversity trade-off at
  deployment time, and even further boost performance. We conduct extensive
  experiments on the MusicLM (Agostinelli et al., 2023) text-to-music generative
  model, where our approach surpasses CFG in terms of quality-diversity Pareto
  optimality. According to human evaluators, our finetuned-then-merged model
  generates samples with higher quality-diversity than the base model augmented
  with CFG. Explore our generations at
  https://google-research.github.io/seanet/musiclm/diverse_music/.
paper_ai_summary: A new finetuning procedure called diversity-rewarded CFG distillation enhances generative models by balancing quality and diversity without inference overhead.
paper_ai_keywords: Classifier-Free Guidance (CFG) distillation objective RL objective diversity reward model merging quality-diversity Pareto optimality MusicLM