To build the video-music CLIP model, we adopt the de-
sign choice in [46], the state-of-the-art video-music retrieval
model. Specifically, we first split the input music and video
into fixed-length segments and use CLIP [36] and music
tagging model [51] to extract visual and audio features sepa-
rately. Given the extracted features, we adopt a transformer
encoder as the video encoder and music encoder to explore
contextual relations and learn a joint multi-modal embedding
space. The model is trained with the InfoNCE contrastive
loss [4] to map positive video-music pairs closer while push-
ing negative pairs further in the CLIP-based joint embedding
space. Loss of videos v to music pieces m is defined as:

Lv→m = − N(cid:88)

L(cid:88)

(cid:34)

i

l

(cid:80)N

j

(cid:80)L

log

exp [s(vi,l, mi,l)/τ ]

k exp [s(vi,l, mj,k)/τ ]

(cid:35)

,
VideoBackgroundMusicGeneration