ging tasks in the MU-LLaMA work [47], is chosen as our
music encoder. The feature representations obtained from
the input encoders of different modalities are then fed into
their respective understanding adaptors. The LLaMA 2
model [65] comprehends these modality signals and input
intentions to carry out downstream tasks. For the music
generation task, we explore and compare two music de-
coders, which are AudioLDM 2 [46] and MusicGen [9],
while music understanding is directly addressed by the
LLaMA 2 model.
In the realm of LLM-assisted music understanding and
generation, there is a notable scarcity of readily available
training data. The MusicCaps dataset [1], which stands
as the largest publicly available dataset for text-to-music
generation, only comprises approximately 28.52 hours of
music accompanied by captions. Moreover, there is a
dearth of datasets for the training of image/video-to-music
generation. Hence, in order to tackle this data scarcity is-
M2UGen