Audio Conditioning for Music Generation via Discrete Bottleneck Features
<|reference_start|>Audio Conditioning for Music Generation via Discrete Bottleneck Features: While most music generation models use textual or parametric conditioning (e.g. tempo, harmony, musical genre), we propose to condition a language model based music generation system with audio input. Our exploration involves two distinct strategies. The first strategy, termed textual inversion, leverages a pre-trained text-to-music model to map audio input to corresponding "pseudowords" in the textual embedding space. For the second model we train a music language model from scratch jointly with a text conditioner and a quantized audio feature extractor. At inference time, we can mix textual and audio conditioning and balance them thanks to a novel double classifier free guidance method. We conduct automatic and human studies that validates our approach. We will release the code and we provide music samples on https://musicgenstyle.github.io in order to show the quality of our model.<|reference_end|>
@article{rouard2024audio,
    title={Audio Conditioning for Music Generation via Discrete Bottleneck Features},
    author={Simon Rouard, Yossi Adi, Jade Copet, Axel Roebel, Alexandre D'efossez},
    journal={arXiv preprint arXiv:2407.12563},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.12563},
    primaryClass={cs.SD eess.AS}
}