title: MusicLDM: Enhancing Novelty in Text-to-Music Generation Using
    Beat-Synchronous Mixup Strategies
summary: Diffusion models have shown promising results in cross-modal generation
  tasks, including text-to-image and text-to-audio generation. However,
  generating music, as a special type of audio, presents unique challenges due to
  limited availability of music data and sensitive issues related to copyright
  and plagiarism. In this paper, to tackle these challenges, we first construct a
  state-of-the-art text-to-music model, MusicLDM, that adapts Stable Diffusion
  and AudioLDM architectures to the music domain. We achieve this by retraining
  the contrastive language-audio pretraining model (CLAP) and the Hifi-GAN
  vocoder, as components of MusicLDM, on a collection of music data samples.
  Then, to address the limitations of training data and to avoid plagiarism, we
  leverage a beat tracking model and propose two different mixup strategies for
  data augmentation: beat-synchronous audio mixup and beat-synchronous latent
  mixup, which recombine training audio directly or via a latent embeddings
  space, respectively. Such mixup strategies encourage the model to interpolate
  between musical training samples and generate new music within the convex hull
  of the training data, making the generated music more diverse while still
  staying faithful to the corresponding style. In addition to popular evaluation
  metrics, we design several new evaluation metrics based on CLAP score to
  demonstrate that our proposed MusicLDM and beat-synchronous mixup strategies
  improve both the quality and novelty of generated music, as well as the
  correspondence between input text and generated music.
paper_authors._id: 64cc5a70eed22517ae0d4db1
paper_authors.name: Ke Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-17T19:21:44.450Z
paper_authors._id: 63fe457052441fe3e874ee22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677608254704-noauth.jpeg
paper_authors.fullname: Ke Chen
paper_authors.type: user
paper_authors.user: RSNuts
paper_authors._id: 64cc5a70eed22517ae0d4db2
paper_authors.name: Yusong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:45:31.178Z
paper_authors._id: 61c3239277d54877f764328e
paper_authors.avatarUrl: /avatars/2d2894cb3d61942cf8661c78350f3c3e.svg
paper_authors.fullname: Yusong Wu
paper_authors.type: user
paper_authors.user: lukewys
paper_authors._id: 64cc5a70eed22517ae0d4db3
paper_authors.name: Haohe Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:45:14.771Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 64cc5a70eed22517ae0d4db4
paper_authors.name: Marianna Nezhurina
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-04T12:44:52.422Z
paper_authors._id: 62500f361684e0335e527bc6
paper_authors.avatarUrl: /avatars/8d2bc4c9cfa8ea82049196431cc3ebea.svg
paper_authors.fullname: Marianna Nezhurina
paper_authors.type: user
paper_authors.user: marianna13
paper_authors._id: 64cc5a70eed22517ae0d4db5
paper_authors.name: Taylor Berg-Kirkpatrick
paper_authors._id: 64cc5a70eed22517ae0d4db6
paper_authors.name: Shlomo Dubnov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T16:15:20.366Z
paper_authors._id: 643da2c837ca9134b589099a
paper_authors.avatarUrl: /avatars/67c317d2b1644b127c4eed53da301ad9.svg
paper_authors.fullname: Shlomo Dubnov
paper_authors.type: user
paper_authors.user: sdubnov
paper_title: MusicLDM: Enhancing Novelty in Text-to-Music Generation Using
    Beat-Synchronous Mixup Strategies
paper_summary: Diffusion models have shown promising results in cross-modal generation
  tasks, including text-to-image and text-to-audio generation. However,
  generating music, as a special type of audio, presents unique challenges due to
  limited availability of music data and sensitive issues related to copyright
  and plagiarism. In this paper, to tackle these challenges, we first construct a
  state-of-the-art text-to-music model, MusicLDM, that adapts Stable Diffusion
  and AudioLDM architectures to the music domain. We achieve this by retraining
  the contrastive language-audio pretraining model (CLAP) and the Hifi-GAN
  vocoder, as components of MusicLDM, on a collection of music data samples.
  Then, to address the limitations of training data and to avoid plagiarism, we
  leverage a beat tracking model and propose two different mixup strategies for
  data augmentation: beat-synchronous audio mixup and beat-synchronous latent
  mixup, which recombine training audio directly or via a latent embeddings
  space, respectively. Such mixup strategies encourage the model to interpolate
  between musical training samples and generate new music within the convex hull
  of the training data, making the generated music more diverse while still
  staying faithful to the corresponding style. In addition to popular evaluation
  metrics, we design several new evaluation metrics based on CLAP score to
  demonstrate that our proposed MusicLDM and beat-synchronous mixup strategies
  improve both the quality and novelty of generated music, as well as the
  correspondence between input text and generated music.
paper_ai_summary: A text-to-music model named MusicLDM, based on Stable Diffusion and AudioLDM, overcomes limitations in music data and plagiarism with beat tracking and mixup strategies, enhancing music quality, novelty, and text correspondence.
paper_ai_keywords: diffusion models text-to-music MusicLDM Stable Diffusion AudioLDM contrastive language-audio pretraining model CLAP Hifi-GAN beat tracking beat-synchronous audio mixup beat-synchronous latent mixup CLAP score