21

major. Finally, this concatenated vector is passed through an embedding layer,

yielding a final input embedding vector with a dimensionality of 512. This en-

riched embedding vector becomes part of the input for training our Transformer

decoder, together with the video embedding vector, to generate sequences that

match with the video’s content. This entire process is represented by the equa-

tion:

Inputt

music = P E(Echord(concat(k, Eq(C t

q) + Er(C t

r))))

(3)

where Inputt

music represents the input music vector at a given time t, concat()

represents the concatenation function, and P E() represents the positional en-

coding function which is used to inject information about the position or order

of elements in a sequence into the representation of those elements. The k rep-

resents the key vector that has a value of either 0 (minor) or 1 (major), C t
q,

C t

r represents the one-hot chord type vector and one-hot chord root vector at
Video2Music