Directly using raw video frames as conditional input is
difficult for model to learn the correspondence between two
different modalities. Thus, it is important to design and
extract meaningful features from video as intermediate rep-
resentations to simplify the learning process. Considering
the style and rhythm relationship between music and video,
we extract semantic, color, and motion features separately to
guide the music generation model.
Semantic Features. We use pretrained CLIP2Video [13]
as the extractor to encode raw video frames into semantic
feature tokens without finetuning. The CLIP2Video model
builds upon the CLIP encoder [36], which is pretrained on
billions of image-text pairs, and further uses a temporal dif-
ference block to learn the temporal context across frames.
The extracted features are supposed to contain representa-
tions of different video semantics, e.g., scenery, sports, and
crowds, which are closely related to the content of music.
VideoBackgroundMusicGeneration