the note density and loudness features. These features are used to construct

a biGRU-based regression model for post-processing, which is able to estimate

note density and loudness from video features. This mechanism allows for the

generation of music with varying rhythms and volume levels.

The core of our proposed Video2Music framework is a novel Affective Multi-

modal Transformer (AMT) model, which generates chords given a video. This

model consists of two fundamental components: an encoder, which takes the

extracted input features from the video as a conditioning factor, and a decoder

4

which takes input features associated with chords and keys extracted from the

audio during training as well as conditioning from the decoder, and learns to

predict new chords during inference. We have set up an extensive experiment,

including an objective experiment, as well as a subjective listening study, which
Video2Music