†Corresponding author.

address this gap, we introduce a Multi-modal Music
Understanding and Generation (M2UGen) framework
that integrates LLM’s abilities to comprehend and gen-
erate music for different modalities. The M2UGen frame-
work is purpose-built to unlock creative potential from
diverse sources of inspiration, encompassing music, im-
age and video through the use of pretrained MERT, ViT,
and ViViT models, respectively. To enable music gen-
eration, we explore the use of AudioLDM 2 and Mu-
sicGen. Bridging multi-modal understanding and music
generation is accomplished through the integration of the
LLaMA 2 model. Furthermore, we make use of the MU-

M2UGen

A PREPRINT
M2UGen