title: MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music
    Generation
summary: Existing text-to-music models can produce high-quality audio with great
  diversity. However, textual prompts alone cannot precisely control temporal
  musical features such as chords and rhythm of the generated music. To address
  this challenge, we introduce MusiConGen, a temporally-conditioned
  Transformer-based text-to-music model that builds upon the pretrained MusicGen
  framework. Our innovation lies in an efficient finetuning mechanism, tailored
  for consumer-grade GPUs, that integrates automatically-extracted rhythm and
  chords as the condition signal. During inference, the condition can either be
  musical features extracted from a reference audio signal, or be user-defined
  symbolic chord sequence, BPM, and textual prompts. Our performance evaluation
  on two datasets -- one derived from extracted features and the other from
  user-created inputs -- demonstrates that MusiConGen can generate realistic
  backing track music that aligns well with the specified conditions. We
  open-source the code and model checkpoints, and provide audio examples online,
  https://musicongen.github.io/musicongen_demo/.
paper_authors._id: 669f3ac8c13da2bbe275f8df
paper_authors.name: Yun-Han Lan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:19.513Z
paper_authors._id: 64f80a0ece75bb0fb5f6cd17
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5Yq5Zjoam_p0-t97bBqRf.jpeg
paper_authors.fullname: 藍雲瀚
paper_authors.type: user
paper_authors.user: Cyan0731
paper_authors._id: 669f3ac8c13da2bbe275f8e0
paper_authors.name: Wen-Yi Hsiao
paper_authors._id: 669f3ac8c13da2bbe275f8e1
paper_authors.name: Hao-Chung Cheng
paper_authors._id: 669f3ac8c13da2bbe275f8e2
paper_authors.name: Yi-Hsuan Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T07:27:17.328Z
paper_authors._id: 66a07ec81bb371390c6414a3
paper_authors.avatarUrl: /avatars/396baf7bd417d8620e09bb8749eaa034.svg
paper_authors.fullname: Yi-Hsuan Yang
paper_authors.type: user
paper_authors.user: yhyangtw
paper_title: MusiConGen: Rhythm and Chord Control for Transformer-Based Text-to-Music
    Generation
paper_summary: Existing text-to-music models can produce high-quality audio with great
  diversity. However, textual prompts alone cannot precisely control temporal
  musical features such as chords and rhythm of the generated music. To address
  this challenge, we introduce MusiConGen, a temporally-conditioned
  Transformer-based text-to-music model that builds upon the pretrained MusicGen
  framework. Our innovation lies in an efficient finetuning mechanism, tailored
  for consumer-grade GPUs, that integrates automatically-extracted rhythm and
  chords as the condition signal. During inference, the condition can either be
  musical features extracted from a reference audio signal, or be user-defined
  symbolic chord sequence, BPM, and textual prompts. Our performance evaluation
  on two datasets -- one derived from extracted features and the other from
  user-created inputs -- demonstrates that MusiConGen can generate realistic
  backing track music that aligns well with the specified conditions. We
  open-source the code and model checkpoints, and provide audio examples online,
  https://musicongen.github.io/musicongen_demo/.
paper_ai_summary: MusiConGen, a Transformer-based text-to-music model, generates high-quality audio by incorporating temporal conditions like rhythm and chords, enhancing control over music generation.
paper_ai_keywords: temporally-conditioned Transformer-based MusicGen finetuning chord sequence BPM audio features user-defined