title: M^{2}UGen: Multi-modal Music Understanding and Generation with the
    Power of Large Language Models
summary: The current landscape of research leveraging large language models (LLMs) is
  experiencing a surge. Many works harness the powerful reasoning capabilities of
  these models to comprehend various modalities, such as text, speech, images,
  videos, etc. They also utilize LLMs to understand human intention and generate
  desired outputs like images, videos, and music. However, research that combines
  both understanding and generation using LLMs is still limited and in its
  nascent stage. To address this gap, we introduce a Multi-modal Music
  Understanding and Generation (M^{2}UGen) framework that integrates LLM's
  abilities to comprehend and generate music for different modalities. The
  M^{2}UGen framework is purpose-built to unlock creative potential from
  diverse sources of inspiration, encompassing music, image, and video through
  the use of pretrained MERT, ViT, and ViViT models, respectively. To enable
  music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging
  multi-modal understanding and music generation is accomplished through the
  integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA
  model to generate extensive datasets that support text/image/video-to-music
  generation, facilitating the training of our M^{2}UGen framework. We conduct
  a thorough evaluation of our proposed framework. The experimental results
  demonstrate that our model achieves or surpasses the performance of the current
  state-of-the-art models.
paper_authors._id: 655c295c973f30a0a593ed04
paper_authors.name: Atin Sakkeer Hussain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T14:46:27.677Z
paper_authors._id: 61b4db3d8c3a7fcce3c71a25
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61b4db3d8c3a7fcce3c71a25/7I035TZlmlTFHtsut92rG.jpeg
paper_authors.fullname: Atin Sakkeer Hussain
paper_authors.type: user
paper_authors.user: crypto-code
paper_authors._id: 655c295c973f30a0a593ed05
paper_authors.name: Shansong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-21T16:38:20.422Z
paper_authors._id: 64c1f3560a2545084f08350b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c1f3560a2545084f08350b/RbSzN5eFvExrfRLX1Xi_g.jpeg
paper_authors.fullname: Shansong Liu
paper_authors.type: user
paper_authors.user: dadinghh2
paper_authors._id: 655c295c973f30a0a593ed06
paper_authors.name: Chenshuo Sun
paper_authors._id: 655c295c973f30a0a593ed07
paper_authors.name: Ying Shan
paper_title: M^{2}UGen: Multi-modal Music Understanding and Generation with the
    Power of Large Language Models
paper_summary: The current landscape of research leveraging large language models (LLMs) is
  experiencing a surge. Many works harness the powerful reasoning capabilities of
  these models to comprehend various modalities, such as text, speech, images,
  videos, etc. They also utilize LLMs to understand human intention and generate
  desired outputs like images, videos, and music. However, research that combines
  both understanding and generation using LLMs is still limited and in its
  nascent stage. To address this gap, we introduce a Multi-modal Music
  Understanding and Generation (M^{2}UGen) framework that integrates LLM's
  abilities to comprehend and generate music for different modalities. The
  M^{2}UGen framework is purpose-built to unlock creative potential from
  diverse sources of inspiration, encompassing music, image, and video through
  the use of pretrained MERT, ViT, and ViViT models, respectively. To enable
  music generation, we explore the use of AudioLDM 2 and MusicGen. Bridging
  multi-modal understanding and music generation is accomplished through the
  integration of the LLaMA 2 model. Furthermore, we make use of the MU-LLaMA
  model to generate extensive datasets that support text/image/video-to-music
  generation, facilitating the training of our M^{2}UGen framework. We conduct
  a thorough evaluation of our proposed framework. The experimental results
  demonstrate that our model achieves or surpasses the performance of the current
  state-of-the-art models.
paper_ai_summary: A framework integrating large language models for multi-modal music understanding and generation outperforms current state-of-the-art models.
paper_ai_keywords: Multi-modal Music Understanding and Generation M$^{2}$UGen pretrained MERT ViT ViViT AudioLDM 2 MusicGen LLaMA 2 MU-LLaMA text/image/video-to-music generation