Multi-modal music generation from images/videos is a
crucial ability of our M2UGen model. CoDi [61] is an
any-to-any generation model, thus it is involved for both
image-to-music (I2M) and video-to-music (V2M) gener-
ation experiments (Tables 4 and 5), while CMT [11] is
for video-to-music generation. The evaluation sets for
I2M and V2M consist of 2,500 pairs of image-music and
video-music, respectively.
In these two sets of experi-
ments, apart from FAD and KL, we introduce a new eval-
uation metric called ImageBind Ranking (IB Rank) [23]
to assess the alignment between the image/video modality
and the generated music. Specifically, we use the Image-
Bind model to obtain embeddings for the images/videos
and the generated music, and then calculate their similar-
ity scores for ranking purposes.
From Tables 4 and 5, it can be seen that our M2UGen
model demonstrates exceptional capabilities in multi-
modal music generation, both in terms of the quality of
M2UGen