Chord TransformerMelodyTransformerAccompaniment TransformerChordMelodyMusicMusic GeneratorFusion EncoderColorEncoderSemanticEncoderTiming EncodingColor HistogramCLIP FeatureTempoEstimatorTempo EmbeddingVideoRGB-DiffFramesVideo ControllerAccompanimentRhythmBarBeatChordRootQual.ChordAttr.RhythmBarBeatNotePitchDura.NotePitchDura.Time10011CC#DD#A#ChromaRoot-QualityAttr.Chord Attributes0B(a) Music Representation123456(b) Pipeline of V-MusProdFramesof the melody xm and the accompaniment xa, ⊕ represents
merging the two parts into a single track.
Music Representation. Symbolic music comprises a set
of music attributes. To encode the dependencies among
different attributes, we design an event-based music rep-
resentation inspired by [21, 38]. We define three types of
tokens, namely Note, Rhythm, and Chord, as the red, yellow,
and blue columns in Fig. 4 (a), respectively. Each token is a
stack of attributes. In particular, the Rhythm token comprises
VideoBackgroundMusicGeneration