title: MusicRL: Aligning Music Generation to Human Preferences
summary: We propose MusicRL, the first music generation system finetuned from human
  feedback. Appreciation of text-to-music models is particularly subjective since
  the concept of musicality as well as the specific intention behind a caption
  are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a
  retro guitar solo or a techno pop beat). Not only this makes supervised
  training of such models challenging, but it also calls for integrating
  continuous human feedback in their post-deployment finetuning. MusicRL is a
  pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete
  audio tokens finetuned with reinforcement learning to maximise sequence-level
  rewards. We design reward functions related specifically to text-adherence and
  audio quality with the help from selected raters, and use those to finetune
  MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial
  dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning
  from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model
  that incorporates human feedback at scale. Human evaluations show that both
  MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU
  combines the two approaches and results in the best model according to human
  raters. Ablation studies shed light on the musical attributes influencing human
  preferences, indicating that text adherence and quality only account for a part
  of it. This underscores the prevalence of subjectivity in musical appreciation
  and calls for further involvement of human listeners in the finetuning of music
  generation models.
paper_authors._id: 65c30546047c54295aba2855
paper_authors.name: Geoffrey Cideron
paper_authors._id: 65c30546047c54295aba2856
paper_authors.name: Sertan Girgin
paper_authors._id: 65c30546047c54295aba2857
paper_authors.name: Mauro Verzetti
paper_authors._id: 65c30546047c54295aba2858
paper_authors.name: Damien Vincent
paper_authors._id: 65c30546047c54295aba2859
paper_authors.name: Matej Kastelic
paper_authors._id: 65c30546047c54295aba285a
paper_authors.name: Zalán Borsos
paper_authors._id: 65c30546047c54295aba285b
paper_authors.name: Brian McWilliams
paper_authors._id: 65c30546047c54295aba285c
paper_authors.name: Victor Ungureanu
paper_authors._id: 65c30546047c54295aba285d
paper_authors.name: Olivier Bachem
paper_authors._id: 65c30546047c54295aba285e
paper_authors.name: Olivier Pietquin
paper_authors._id: 65c30546047c54295aba285f
paper_authors.name: Matthieu Geist
paper_authors._id: 65c30546047c54295aba2860
paper_authors.name: Léonard Hussenot
paper_authors._id: 65c30546047c54295aba2861
paper_authors.name: Neil Zeghidour
paper_authors._id: 65c30546047c54295aba2862
paper_authors.name: Andrea Agostinelli
paper_title: MusicRL: Aligning Music Generation to Human Preferences
paper_summary: We propose MusicRL, the first music generation system finetuned from human
  feedback. Appreciation of text-to-music models is particularly subjective since
  the concept of musicality as well as the specific intention behind a caption
  are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a
  retro guitar solo or a techno pop beat). Not only this makes supervised
  training of such models challenging, but it also calls for integrating
  continuous human feedback in their post-deployment finetuning. MusicRL is a
  pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete
  audio tokens finetuned with reinforcement learning to maximise sequence-level
  rewards. We design reward functions related specifically to text-adherence and
  audio quality with the help from selected raters, and use those to finetune
  MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial
  dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning
  from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model
  that incorporates human feedback at scale. Human evaluations show that both
  MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU
  combines the two approaches and results in the best model according to human
  raters. Ablation studies shed light on the musical attributes influencing human
  preferences, indicating that text adherence and quality only account for a part
  of it. This underscores the prevalence of subjectivity in musical appreciation
  and calls for further involvement of human listeners in the finetuning of music
  generation models.
paper_ai_keywords: autoregressive MusicLM discrete audio tokens reinforcement learning sequence-level rewards reinforcement learning from human feedback (RLHF) text-adherence audio quality pairwise preferences human evaluations ablation studies text-to-music model