title: Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of
    Experts
summary: As the research of Multimodal Large Language Models (MLLMs) becomes popular,
  an advancing MLLM model is typically required to handle various textual and
  visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for
  real-world applications. However, due to the significant differences in
  representation and distribution among data from various tasks, simply mixing
  data of all tasks together leads to the well-known``multi-task conflict" issue,
  resulting in performance degradation across various tasks. To address this
  issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture
  suitable for MLLM, which acquires the multi-task capabilities through multiple
  sparsely activated experts. To speed up the training and inference of
  Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation
  (LoRA) structure. Extensive experiments on multiple latest benchmarks
  demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are
  released in our Project Page: https://github.com/MetabrainAGI/Awaker.
paper_authors._id: 673c25ff11b7efeeed69461d
paper_authors.name: Jinqiang Long
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-20T10:09:08.488Z
paper_authors._id: 6225f8edb471983f28f8fc44
paper_authors.avatarUrl: /avatars/00ca46b8a745540b2ec1cb135fe02194.svg
paper_authors.fullname: Long
paper_authors.type: user
paper_authors.user: seventeen
paper_authors._id: 673c25ff11b7efeeed69461e
paper_authors.name: Yanqi Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:23:30.039Z
paper_authors._id: 66cde57cb1fe4c78fe3ab770
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66cde57cb1fe4c78fe3ab770/0R1aA-f_XLjCfy1HwqZ-p.jpeg
paper_authors.fullname: Yanqi Dai
paper_authors.type: user
paper_authors.user: YanqiDai
paper_authors._id: 673c25ff11b7efeeed69461f
paper_authors.name: Guoxing Yang
paper_authors._id: 673c25ff11b7efeeed694620
paper_authors.name: Hongpeng Lin
paper_authors._id: 673c25ff11b7efeeed694621
paper_authors.name: Nanyi Fei
paper_authors._id: 673c25ff11b7efeeed694622
paper_authors.name: Yizhao Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:23:56.495Z
paper_authors._id: 661c96f48921f03a9dae04c3
paper_authors.avatarUrl: /avatars/c486a45ffea7d1a8c72bd8512014b07e.svg
paper_authors.fullname: Yizhao Gao
paper_authors.type: user
paper_authors.user: LongMountain
paper_authors._id: 673c25ff11b7efeeed694623
paper_authors.name: Zhiwu Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-19T15:23:50.216Z
paper_authors._id: 6351e3035b904878f1d36719
paper_authors.avatarUrl: /avatars/a7b3a5faacaa483fab1a5b1b95bcc0a4.svg
paper_authors.fullname: Zhiwu Lu
paper_authors.type: user
paper_authors.user: zewlu
paper_title: Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of
    Experts
paper_summary: As the research of Multimodal Large Language Models (MLLMs) becomes popular,
  an advancing MLLM model is typically required to handle various textual and
  visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for
  real-world applications. However, due to the significant differences in
  representation and distribution among data from various tasks, simply mixing
  data of all tasks together leads to the well-known``multi-task conflict" issue,
  resulting in performance degradation across various tasks. To address this
  issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture
  suitable for MLLM, which acquires the multi-task capabilities through multiple
  sparsely activated experts. To speed up the training and inference of
  Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation
  (LoRA) structure. Extensive experiments on multiple latest benchmarks
  demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are
  released in our Project Page: https://github.com/MetabrainAGI/Awaker.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) VQA Detection OCR ChartQA multi-task conflict Mixture of Experts (MoE) sparsely activated experts low-rank adaptation (LoRA)