Multimodal Music Generation with Explicit Bridges and Retrieval
  Augmentation
Multimodal music generation aims to produce music from diverse input
modalities, including text, videos, and images. Existing methods use a common
embedding space for multimodal fusion. Despite their effectiveness in other
modalities, their application in multimodal music generation faces challenges
of data scarcity, weak cross-modal alignment, and limited controllability. This
paper addresses these issues by using explicit bridges of text and music for
multimodal alignment. We introduce a novel method named Visuals Music Bridge
(VMB). Specifically, a Multimodal Music Description Model converts visual
inputs into detailed textual descriptions to provide the text bridge; a
Dual-track Music Retrieval module that combines broad and targeted retrieval
strategies to provide the music bridge and enable user control. Finally, we
design an Explicitly Conditioned Music Generation framework to generate music
based on the two bridges. We conduct experiments on video-to-music,
image-to-music, text-to-music, and controllable music generation tasks, along
with experiments on controllability. The results demonstrate that VMB
significantly enhances music quality, modality, and customization alignment
compared to previous methods. VMB sets a new standard for interpretable and
expressive multimodal music generation with applications in various multimedia
fields. Demos and code are available at https://github.com/wbs2788/VMB.
675fc58aa733b7f21793a3c9
Baisen Wang
admin_assigned
2024-12-16T10:17:48.195Z
6433a54982ca403c44e0560c
/avatars/66ca5c1ba2351909b1b8d3a2a774e186.svg
baisen wang
user
wbs2788
675fc58aa733b7f21793a3ca
Le Zhuo
admin_assigned
2024-12-16T10:17:53.867Z
6358a167f56b03ec9147074d
/avatars/e54ea7bf0c240cf76d538296efb3976c.svg
Le Zhuo
user
JackyZhuo
675fc58aa733b7f21793a3cb
Zhaokai Wang
claimed_verified
2024-12-16T09:16:18.569Z
665d4b515fdfe8f923e347a7
/avatars/d114b24c02dadfca0a8aee104755a8ec.svg
Zhaokai Wang
user
wzk1015
675fc58aa733b7f21793a3cc
Chenxi Bao
admin_assigned
2024-12-16T10:18:05.888Z
64265b32ad1e3b0e6e91d009
/avatars/f1f9971bf804eb1665d157fc3d9b5209.svg
chenxi
user
baochenxi
675fc58aa733b7f21793a3cd
Wu Chengjing
675fc58aa733b7f21793a3ce
Xuecheng Nie
675fc58aa733b7f21793a3cf
Jiao Dai
675fc58aa733b7f21793a3d0
Jizhong Han
675fc58aa733b7f21793a3d1
Yue Liao
admin_assigned
2024-12-16T10:18:50.052Z
670f827bb94a3734d270f707
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/D6qCPBMJAUgozfG7YTwky.png
Yue Liao
user
morninghaze
675fc58aa733b7f21793a3d2
Si Liu
Multimodal Music Generation with Explicit Bridges and Retrieval
  Augmentation
Multimodal music generation aims to produce music from diverse input
modalities, including text, videos, and images. Existing methods use a common
embedding space for multimodal fusion. Despite their effectiveness in other
modalities, their application in multimodal music generation faces challenges
of data scarcity, weak cross-modal alignment, and limited controllability. This
paper addresses these issues by using explicit bridges of text and music for
multimodal alignment. We introduce a novel method named Visuals Music Bridge
(VMB). Specifically, a Multimodal Music Description Model converts visual
inputs into detailed textual descriptions to provide the text bridge; a
Dual-track Music Retrieval module that combines broad and targeted retrieval
strategies to provide the music bridge and enable user control. Finally, we
design an Explicitly Conditioned Music Generation framework to generate music
based on the two bridges. We conduct experiments on video-to-music,
image-to-music, text-to-music, and controllable music generation tasks, along
with experiments on controllability. The results demonstrate that VMB
significantly enhances music quality, modality, and customization alignment
compared to previous methods. VMB sets a new standard for interpretable and
expressive multimodal music generation with applications in various multimedia
fields. Demos and code are available at https://github.com/wbs2788/VMB.
A new method for multimodal music generation uses explicit text and music bridges to enhance quality, modality, and customization compared to previous approaches.
Multimodal Music Description Model Dual-track Music Retrieval Explicitly Conditioned Music Generation video-to-music image-to-music text-to-music controllable music generation