MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music
  Generation
Existing text-to-music models can produce high-quality audio with great
diversity. However, textual prompts alone cannot precisely control temporal
musical features such as chords and rhythm of the generated music. To address
this challenge, we introduce MusiConGen, a temporally-conditioned
Transformer-based text-to-music model that builds upon the pretrained MusicGen
framework. Our innovation lies in an efficient finetuning mechanism, tailored
for consumer-grade GPUs, that integrates automatically-extracted rhythm and
chords as the condition signal. During inference, the condition can either be
musical features extracted from a reference audio signal, or be user-defined
symbolic chord sequence, BPM, and textual prompts. Our performance evaluation
on two datasets -- one derived from extracted features and the other from
user-created inputs -- demonstrates that MusiConGen can generate realistic
backing track music that aligns well with the specified conditions. We
open-source the code and model checkpoints, and provide audio examples online,
https://musicongen.github.io/musicongen_demo/.
669f3ac8c13da2bbe275f8df
Yun-Han Lan
claimed_verified
2024-07-24T07:27:19.513Z
64f80a0ece75bb0fb5f6cd17
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5Yq5Zjoam_p0-t97bBqRf.jpeg
藍雲瀚
user
Cyan0731
669f3ac8c13da2bbe275f8e0
Wen-Yi Hsiao
669f3ac8c13da2bbe275f8e1
Hao-Chung Cheng
669f3ac8c13da2bbe275f8e2
Yi-Hsuan Yang
claimed_verified
2024-07-24T07:27:17.328Z
66a07ec81bb371390c6414a3
/avatars/396baf7bd417d8620e09bb8749eaa034.svg
Yi-Hsuan Yang
user
yhyangtw
MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music
  Generation
Existing text-to-music models can produce high-quality audio with great
diversity. However, textual prompts alone cannot precisely control temporal
musical features such as chords and rhythm of the generated music. To address
this challenge, we introduce MusiConGen, a temporally-conditioned
Transformer-based text-to-music model that builds upon the pretrained MusicGen
framework. Our innovation lies in an efficient finetuning mechanism, tailored
for consumer-grade GPUs, that integrates automatically-extracted rhythm and
chords as the condition signal. During inference, the condition can either be
musical features extracted from a reference audio signal, or be user-defined
symbolic chord sequence, BPM, and textual prompts. Our performance evaluation
on two datasets -- one derived from extracted features and the other from
user-created inputs -- demonstrates that MusiConGen can generate realistic
backing track music that aligns well with the specified conditions. We
open-source the code and model checkpoints, and provide audio examples online,
https://musicongen.github.io/musicongen_demo/.
MusiConGen, a Transformer-based text-to-music model, generates high-quality audio by incorporating temporal conditions like rhythm and chords, enhancing control over music generation.
temporally-conditioned Transformer-based MusicGen finetuning chord sequence BPM audio features user-defined