  Recent studies have demonstrated Large Language Models (LLMs) can extend
their zero-shot generalization capabilities to multimodal learning through
instruction tuning. As more modalities and downstream tasks are introduced,
negative conflicts and interference may have a worse impact on performance.
While this phenomenon has been overlooked in previous work, we propose a novel
and extensible framework, called \mname, for comprehensive studies and
experimentation on multimodal learning with Multimodal Large Language Models
(MLLMs). Specifically, we combine the well-known Mixture-of-Experts (MoE) and
one of the representative PEFT techniques, \emph{i.e.,} LoRA, designing a novel
LLM-based decoder, called LoRA-MoE, for multimodal learning. The experimental
results (about 20\% improvement) have shown the effectiveness and versatility
of our design in various 2D and 3D downstream tasks. Code and corresponding
dataset will be available soon.

Computer Vision and Pattern Recognition