15

Figure 5: Top 30 keys in our dataset (before chord normalization).

each second of video to extract the below features. The extraction process for

each of these features is described below.

3.2.1. Semantic features

We harnessed the capabilities of CLIP (Contrastive Language-Image Pre-

training) (Radford et al., 2021), a powerful pretrained model, as a feature ex-

tractor. This model enabled us to encode the raw video frames into semantic

feature tokens without the need for fine-tuning. We utilized CLIP to extract la-

tent features from each video frame. These extracted latent features encompass

a wide range of video semantics, including serene beach scenes, adventurous

outdoor activities, and bustling city streets.

3.2.2. Emotion

To estimate the emotions expressed in a (muted) video on a per-second basis,

we employ the CLIP model (Radford et al., 2021). This model has been trained

on an extensive dataset containing 400 million image-text pairs, providing it
Video2Music