title: Multimodal Music Generation with Explicit Bridges and Retrieval
    Augmentation
summary: Multimodal music generation aims to produce music from diverse input
  modalities, including text, videos, and images. Existing methods use a common
  embedding space for multimodal fusion. Despite their effectiveness in other
  modalities, their application in multimodal music generation faces challenges
  of data scarcity, weak cross-modal alignment, and limited controllability. This
  paper addresses these issues by using explicit bridges of text and music for
  multimodal alignment. We introduce a novel method named Visuals Music Bridge
  (VMB). Specifically, a Multimodal Music Description Model converts visual
  inputs into detailed textual descriptions to provide the text bridge; a
  Dual-track Music Retrieval module that combines broad and targeted retrieval
  strategies to provide the music bridge and enable user control. Finally, we
  design an Explicitly Conditioned Music Generation framework to generate music
  based on the two bridges. We conduct experiments on video-to-music,
  image-to-music, text-to-music, and controllable music generation tasks, along
  with experiments on controllability. The results demonstrate that VMB
  significantly enhances music quality, modality, and customization alignment
  compared to previous methods. VMB sets a new standard for interpretable and
  expressive multimodal music generation with applications in various multimedia
  fields. Demos and code are available at https://github.com/wbs2788/VMB.
paper_authors._id: 675fc58aa733b7f21793a3c9
paper_authors.name: Baisen Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:17:48.195Z
paper_authors._id: 6433a54982ca403c44e0560c
paper_authors.avatarUrl: /avatars/66ca5c1ba2351909b1b8d3a2a774e186.svg
paper_authors.fullname: baisen wang
paper_authors.type: user
paper_authors.user: wbs2788
paper_authors._id: 675fc58aa733b7f21793a3ca
paper_authors.name: Le Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:17:53.867Z
paper_authors._id: 6358a167f56b03ec9147074d
paper_authors.avatarUrl: /avatars/e54ea7bf0c240cf76d538296efb3976c.svg
paper_authors.fullname: Le Zhuo
paper_authors.type: user
paper_authors.user: JackyZhuo
paper_authors._id: 675fc58aa733b7f21793a3cb
paper_authors.name: Zhaokai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-16T09:16:18.569Z
paper_authors._id: 665d4b515fdfe8f923e347a7
paper_authors.avatarUrl: /avatars/d114b24c02dadfca0a8aee104755a8ec.svg
paper_authors.fullname: Zhaokai Wang
paper_authors.type: user
paper_authors.user: wzk1015
paper_authors._id: 675fc58aa733b7f21793a3cc
paper_authors.name: Chenxi Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:18:05.888Z
paper_authors._id: 64265b32ad1e3b0e6e91d009
paper_authors.avatarUrl: /avatars/f1f9971bf804eb1665d157fc3d9b5209.svg
paper_authors.fullname: chenxi
paper_authors.type: user
paper_authors.user: baochenxi
paper_authors._id: 675fc58aa733b7f21793a3cd
paper_authors.name: Wu Chengjing
paper_authors._id: 675fc58aa733b7f21793a3ce
paper_authors.name: Xuecheng Nie
paper_authors._id: 675fc58aa733b7f21793a3cf
paper_authors.name: Jiao Dai
paper_authors._id: 675fc58aa733b7f21793a3d0
paper_authors.name: Jizhong Han
paper_authors._id: 675fc58aa733b7f21793a3d1
paper_authors.name: Yue Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:18:50.052Z
paper_authors._id: 670f827bb94a3734d270f707
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/D6qCPBMJAUgozfG7YTwky.png
paper_authors.fullname: Yue Liao
paper_authors.type: user
paper_authors.user: morninghaze
paper_authors._id: 675fc58aa733b7f21793a3d2
paper_authors.name: Si Liu
paper_title: Multimodal Music Generation with Explicit Bridges and Retrieval
    Augmentation
paper_summary: Multimodal music generation aims to produce music from diverse input
  modalities, including text, videos, and images. Existing methods use a common
  embedding space for multimodal fusion. Despite their effectiveness in other
  modalities, their application in multimodal music generation faces challenges
  of data scarcity, weak cross-modal alignment, and limited controllability. This
  paper addresses these issues by using explicit bridges of text and music for
  multimodal alignment. We introduce a novel method named Visuals Music Bridge
  (VMB). Specifically, a Multimodal Music Description Model converts visual
  inputs into detailed textual descriptions to provide the text bridge; a
  Dual-track Music Retrieval module that combines broad and targeted retrieval
  strategies to provide the music bridge and enable user control. Finally, we
  design an Explicitly Conditioned Music Generation framework to generate music
  based on the two bridges. We conduct experiments on video-to-music,
  image-to-music, text-to-music, and controllable music generation tasks, along
  with experiments on controllability. The results demonstrate that VMB
  significantly enhances music quality, modality, and customization alignment
  compared to previous methods. VMB sets a new standard for interpretable and
  expressive multimodal music generation with applications in various multimedia
  fields. Demos and code are available at https://github.com/wbs2788/VMB.
paper_ai_summary: A new method for multimodal music generation uses explicit text and music bridges to enhance quality, modality, and customization compared to previous approaches.
paper_ai_keywords: Multimodal Music Description Model Dual-track Music Retrieval Explicitly Conditioned Music Generation video-to-music image-to-music text-to-music controllable music generation