M(cid:88)

i=0

25

The matching chord type positions of this vector, which has a similar format

to the output vector of the decoder, that belong to the predicted emotion are

activated. These chord type attributes are compared to the generated chord

qualities. Then, yemo

is a i-th element of yemo

i

, and σ(zi) is the sigmoid function,

i

which transforms the logit zi into a value between 0 and 1.

To obtain the chord attributes that match the emotion of the video, we use

the following procedure: first, we use CLIP to obtain a probability for each

of our five emotion categories for the video fragment. The choice of these five

emotions (exciting, fearful, tense, sad, relaxing) for mapping to corresponding

chords is grounded in the MVED dataset (Pandeya et al., 2021) as discussed

in Section 3.2.2. Then, we take the emotion with the highest probability, and

use Table 1 to find the matching chord attributes. If an emotion has multiple
Video2Music