title: MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
summary: For Large Vision-Language Models (LVLMs), scaling the model can effectively
  improve performance. However, expanding model parameters significantly
  increases the training and inferring costs, as all model parameters are
  activated for each token in the calculation. In this work, we propose a novel
  training strategy MoE-tuning for LVLMs, which can constructing a sparse model
  with an outrageous number of parameter but a constant computational cost, and
  effectively addresses the performance degradation typically associated with
  multi-modal learning and model sparsity. Furthermore, we present the MoE-LLaVA
  framework, a MoE-based sparse LVLM architecture. This framework uniquely
  activates only the top-k experts through routers during deployment, keeping the
  remaining experts inactive. Our extensive experiments highlight the excellent
  capabilities of MoE-LLaVA in visual understanding and its potential to reduce
  hallucinations in model outputs. Remarkably, with just 3 billion sparsely
  activated parameters, MoE-LLaVA demonstrates performance comparable to the
  LLaVA-1.5-7B on various visual understanding datasets and even surpasses the
  LLaVA-1.5-13B in object hallucination benchmarks. Through MoE-LLaVA, we aim to
  establish a baseline for sparse LVLMs and provide valuable insights for future
  research in developing more efficient and effective multi-modal learning
  systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.
paper_authors._id: 65b86b41e0bde92c1761c61a
paper_authors.name: Bin Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T08:06:36.700Z
paper_authors._id: 6367a8175bb06007ea099b8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6367a8175bb06007ea099b8f/IjG7HyWyWRlVt_XwRbxRW.jpeg
paper_authors.fullname: linbin
paper_authors.type: user
paper_authors.user: LanguageBind
paper_authors._id: 65b86b41e0bde92c1761c61b
paper_authors.name: Zhenyu Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:15:08.478Z
paper_authors._id: 63c09aa88d1175e3399e906e
paper_authors.avatarUrl: /avatars/a1b8a89e83275f049f7bf75af9086d84.svg
paper_authors.fullname: Zhenyu Tang
paper_authors.type: user
paper_authors.user: royjames
paper_authors._id: 65b86b41e0bde92c1761c61c
paper_authors.name: Yang Ye
paper_authors._id: 65b86b41e0bde92c1761c61d
paper_authors.name: Jiaxi Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-04T10:28:04.520Z
paper_authors._id: 6389a80b4b547a364ea041ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1669965806283-noauth.jpeg
paper_authors.fullname: Jessy Tsui
paper_authors.type: user
paper_authors.user: JessyTsu1
paper_authors._id: 65b86b41e0bde92c1761c61e
paper_authors.name: Bin Zhu
paper_authors._id: 65b86b41e0bde92c1761c61f
paper_authors.name: Peng Jin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-17T11:31:19.940Z
paper_authors._id: 651585ea18cfed0d30bee586
paper_authors.avatarUrl: /avatars/579e468334102472d870875fe40302e6.svg
paper_authors.fullname: Peng Jin
paper_authors.type: user
paper_authors.user: Chat-UniVi
paper_authors._id: 65b86b41e0bde92c1761c620
paper_authors.name: Junwu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:16:15.836Z
paper_authors._id: 64979385e486365ca6b3c030
paper_authors.avatarUrl: /avatars/589344a9db41873d78452560cca3d234.svg
paper_authors.fullname: Junwu Zhang
paper_authors.type: user
paper_authors.user: junwuzhang
paper_authors._id: 65b86b41e0bde92c1761c621
paper_authors.name: Munan Ning
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:16:22.621Z
paper_authors._id: 62a9748708a7ea93ff6d48ab
paper_authors.avatarUrl: /avatars/c6358f81e31f7d31e14899011ce67e09.svg
paper_authors.fullname: Munan Ning
paper_authors.type: user
paper_authors.user: MasonNing
paper_authors._id: 65b86b41e0bde92c1761c622
paper_authors.name: Li Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:12:33.024Z
paper_authors._id: 66135a5e50350afe76beebce
paper_authors.avatarUrl: /avatars/370a4b83949355feb050c2cb0425c264.svg
paper_authors.fullname: yl2488
paper_authors.type: user
paper_authors.user: yl2488
paper_title: MoE-LLaVA: Mixture of Experts for Large Vision-Language Models
paper_summary: For Large Vision-Language Models (LVLMs), scaling the model can effectively
  improve performance. However, expanding model parameters significantly
  increases the training and inferring costs, as all model parameters are
  activated for each token in the calculation. In this work, we propose a novel
  training strategy MoE-tuning for LVLMs, which can constructing a sparse model
  with an outrageous number of parameter but a constant computational cost, and
  effectively addresses the performance degradation typically associated with
  multi-modal learning and model sparsity. Furthermore, we present the MoE-LLaVA
  framework, a MoE-based sparse LVLM architecture. This framework uniquely
  activates only the top-k experts through routers during deployment, keeping the
  remaining experts inactive. Our extensive experiments highlight the excellent
  capabilities of MoE-LLaVA in visual understanding and its potential to reduce
  hallucinations in model outputs. Remarkably, with just 3 billion sparsely
  activated parameters, MoE-LLaVA demonstrates performance comparable to the
  LLaVA-1.5-7B on various visual understanding datasets and even surpasses the
  LLaVA-1.5-13B in object hallucination benchmarks. Through MoE-LLaVA, we aim to
  establish a baseline for sparse LVLMs and provide valuable insights for future
  research in developing more efficient and effective multi-modal learning
  systems. Code is released at https://github.com/PKU-YuanGroup/MoE-LLaVA.
paper_ai_summary: MoE-LLaVA, a sparse Large Vision-Language Model architecture, improves performance and reduces hallucinations with fewer active parameters compared to larger models.
paper_ai_keywords: MoE-tuning LVLMs sparse model computational cost performance degradation multi-modal learning model sparsity MoE-LLaVA routers visual understanding object hallucination