3.5 Training Method

Training a MLLM model from scratch is computation-
ally expensive, which has led several models [40, 58, 71]
to adopt the LoRA fine-tuning approach.
In our train-
ing method, we alleviate the computational burden by
freezing the encoders and generation models, focusing
the training efforts on the multi-modal understanding
adapters and output projector. This strategy not only re-
duces computational costs but also enhances training effi-
ciency. To further simplify the training process and min-
imize the number of trainable parameters, we apply the
LoRA method [31] to train the LLaMA 2 model. In the
training pipeline, we utilize the following loss function to
guide the optimization process:



LCE(ytokens, f (y)logits)
+∥yembeddings − g(f (x)hidden)∥,

if music

Loss =

LCE(ytokens, f (y)logits),

else
M2UGen