In the final training stage, the LoRA training strategy is
employed to train the LLaMA 2 model, concurrently fine-
tuning the Multi-modal Understanding Adapter and Out-
put Projection layer. This stage utilizes datasets including
Alpaca, MusicQA, MUImage, MUVideo, and MUEdit.

12

M2UGen

A PREPRINT

Figure 4: Music Oriented Dataset. Examples from the MUCaps, MUEdit, MUImage and MUVideo datasets used to
train the M2UGen model.

To signal the M2UGen model to generate both music and
text, the output text in MUImage, MUVideo, and MUEdit
datasets is extended with the special audio tokens.

a reduced number of epochs and a smaller dataset during
the final stage of model training.

B.2 Reasoning for Training Strategy
M2UGen