title: Audio Conditioning for Music Generation via Discrete Bottleneck Features
summary: While most music generation models use textual or parametric conditioning
  (e.g. tempo, harmony, musical genre), we propose to condition a language model
  based music generation system with audio input. Our exploration involves two
  distinct strategies. The first strategy, termed textual inversion, leverages a
  pre-trained text-to-music model to map audio input to corresponding
  "pseudowords" in the textual embedding space. For the second model we train a
  music language model from scratch jointly with a text conditioner and a
  quantized audio feature extractor. At inference time, we can mix textual and
  audio conditioning and balance them thanks to a novel double classifier free
  guidance method. We conduct automatic and human studies that validates our
  approach. We will release the code and we provide music samples on
  https://musicgenstyle.github.io in order to show the quality of our model.
paper_authors._id: 66987bdba9778d20ff251561
paper_authors.name: Simon Rouard
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-03T11:58:08.371Z
paper_authors._id: 66fe867df44983580b27700d
paper_authors.avatarUrl: /avatars/d2af73b02e1b361dd9e25dcc83c890a0.svg
paper_authors.fullname: Simon Rouard
paper_authors.type: user
paper_authors.user: simonrouard
paper_authors._id: 66987bdba9778d20ff251562
paper_authors.name: Yossi Adi
paper_authors._id: 66987bdba9778d20ff251563
paper_authors.name: Jade Copet
paper_authors._id: 66987bdba9778d20ff251564
paper_authors.name: Axel Roebel
paper_authors._id: 66987bdba9778d20ff251565
paper_authors.name: Alexandre Défossez
paper_title: Audio Conditioning for Music Generation via Discrete Bottleneck Features
paper_summary: While most music generation models use textual or parametric conditioning
  (e.g. tempo, harmony, musical genre), we propose to condition a language model
  based music generation system with audio input. Our exploration involves two
  distinct strategies. The first strategy, termed textual inversion, leverages a
  pre-trained text-to-music model to map audio input to corresponding
  "pseudowords" in the textual embedding space. For the second model we train a
  music language model from scratch jointly with a text conditioner and a
  quantized audio feature extractor. At inference time, we can mix textual and
  audio conditioning and balance them thanks to a novel double classifier free
  guidance method. We conduct automatic and human studies that validates our
  approach. We will release the code and we provide music samples on
  https://musicgenstyle.github.io in order to show the quality of our model.
paper_ai_summary: A music generation system uses audio input for conditioning, employing both textual inversion and a novel joint training approach with text and audio features, achieving high-quality results.
paper_ai_keywords: text-to-music model pseudowords textual embedding space music language model quantized audio feature extractor double classifier-free guidance method