title: LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
summary: We introduce LLaVA-MoD, a novel framework designed to enable the efficient
  training of small-scale Multimodal Language Models (s-MLLM) by distilling
  knowledge from large-scale MLLM (l-MLLM). Our approach tackles two fundamental
  challenges in MLLM distillation. First, we optimize the network structure of
  s-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the
  language model, striking a balance between computational efficiency and model
  expressiveness. Second, we propose a progressive knowledge transfer strategy to
  ensure comprehensive knowledge migration. This strategy begins with mimic
  distillation, where we minimize the Kullback-Leibler (KL) divergence between
  output distributions to enable the student model to emulate the teacher
  network's understanding. Following this, we introduce preference distillation
  via Direct Preference Optimization (DPO), where the key lies in treating l-MLLM
  as the reference model. During this phase, the s-MLLM's ability to discriminate
  between superior and inferior examples is significantly enhanced beyond l-MLLM,
  leading to a better student that surpasses its teacher, particularly in
  hallucination benchmarks. Extensive experiments demonstrate that LLaVA-MoD
  outperforms existing models across various multimodal benchmarks while
  maintaining a minimal number of activated parameters and low computational
  costs. Remarkably, LLaVA-MoD, with only 2B activated parameters, surpasses
  Qwen-VL-Chat-7B by an average of 8.8% across benchmarks, using merely 0.3% of
  the training data and 23% trainable parameters. These results underscore
  LLaVA-MoD's ability to effectively distill comprehensive knowledge from its
  teacher model, paving the way for the development of more efficient MLLMs. The
  code will be available on: https://github.com/shufangxun/LLaVA-MoD.
paper_authors._id: 66cfdd7c39ce70f23b715ec2
paper_authors.name: Fangxun Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T12:26:05.343Z
paper_authors._id: 6491b1b2c1741666238f8a0f
paper_authors.avatarUrl: /avatars/9246c9ef06d80bd8628426375c95d4be.svg
paper_authors.fullname: JackShu
paper_authors.type: user
paper_authors.user: Shuhuhuhu
paper_authors._id: 66cfdd7c39ce70f23b715ec3
paper_authors.name: Yue Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:11:04.820Z
paper_authors._id: 666a5cd9148d701ff8340e8c
paper_authors.avatarUrl: /avatars/8536b92af349e7b5f22d6b5bd67f77a1.svg
paper_authors.fullname: Yue Liao
paper_authors.type: user
paper_authors.user: yliao
paper_authors._id: 66cfdd7c39ce70f23b715ec4
paper_authors.name: Le Zhuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:11:10.659Z
paper_authors._id: 6358a167f56b03ec9147074d
paper_authors.avatarUrl: /avatars/e54ea7bf0c240cf76d538296efb3976c.svg
paper_authors.fullname: Le Zhuo
paper_authors.type: user
paper_authors.user: JackyZhuo
paper_authors._id: 66cfdd7c39ce70f23b715ec5
paper_authors.name: Chenning Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:11:16.536Z
paper_authors._id: 645c8fe95ebf379fd6d625e4
paper_authors.avatarUrl: /avatars/728d039b9512d404a9048714ec74008b.svg
paper_authors.fullname: chenning xu
paper_authors.type: user
paper_authors.user: cnxu
paper_authors._id: 66cfdd7c39ce70f23b715ec6
paper_authors.name: Guanghao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:11:22.558Z
paper_authors._id: 6662841168009120af048273
paper_authors.avatarUrl: /avatars/dc6e5628d631998874f0b76d59eec051.svg
paper_authors.fullname: Guanghao Zhang
paper_authors.type: user
paper_authors.user: zghao
paper_authors._id: 66cfdd7c39ce70f23b715ec7
paper_authors.name: Haonan Shi
paper_authors._id: 66cfdd7c39ce70f23b715ec8
paper_authors.name: Long Chen
paper_authors._id: 66cfdd7c39ce70f23b715ec9
paper_authors.name: Tao Zhong
paper_authors._id: 66cfdd7c39ce70f23b715eca
paper_authors.name: Wanggui He
paper_authors._id: 66cfdd7c39ce70f23b715ecb
paper_authors.name: Siming Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:57:48.589Z
paper_authors._id: 6485dd6d07a2c1915060f603
paper_authors.avatarUrl: /avatars/8594d647359a7d19ab29b8ec91d1444e.svg
paper_authors.fullname: fu
paper_authors.type: user
paper_authors.user: simingfu
paper_authors._id: 66cfdd7c39ce70f23b715ecc
paper_authors.name: Haoyuan Li
paper_authors._id: 66cfdd7c39ce70f23b715ecd
paper_authors.name: Bolin Li
paper_authors._id: 66cfdd7c39ce70f23b715ece
paper_authors.name: Zhelun Yu
paper_authors._id: 66cfdd7c39ce70f23b715ecf
paper_authors.name: Si Liu
paper_authors._id: 66cfdd7c39ce70f23b715ed0
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T08:12:02.304Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 66cfdd7c39ce70f23b715ed1
paper_authors.name: Hao Jiang
paper_title: LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
paper_summary: We introduce LLaVA-MoD, a novel framework designed to enable the efficient
  training of small-scale Multimodal Language Models (s-MLLM) by distilling
  knowledge from large-scale MLLM (l-MLLM). Our approach tackles two fundamental
  challenges in MLLM distillation. First, we optimize the network structure of
  s-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the
  language model, striking a balance between computational efficiency and model
  expressiveness. Second, we propose a progressive knowledge transfer strategy to
  ensure comprehensive knowledge migration. This strategy begins with mimic
  distillation, where we minimize the Kullback-Leibler (KL) divergence between
  output distributions to enable the student model to emulate the teacher
  network's understanding. Following this, we introduce preference distillation
  via Direct Preference Optimization (DPO), where the key lies in treating l-MLLM
  as the reference model. During this phase, the s-MLLM's ability to discriminate
  between superior and inferior examples is significantly enhanced beyond l-MLLM,
  leading to a better student that surpasses its teacher, particularly in
  hallucination benchmarks. Extensive experiments demonstrate that LLaVA-MoD
  outperforms existing models across various multimodal benchmarks while
  maintaining a minimal number of activated parameters and low computational
  costs. Remarkably, LLaVA-MoD, with only 2B activated parameters, surpasses
  Qwen-VL-Chat-7B by an average of 8.8% across benchmarks, using merely 0.3% of
  the training data and 23% trainable parameters. These results underscore
  LLaVA-MoD's ability to effectively distill comprehensive knowledge from its
  teacher model, paving the way for the development of more efficient MLLMs. The
  code will be available on: https://github.com/shufangxun/LLaVA-MoD.
paper_ai_summary: LLaVA-MoD employs a sparse Mixture of Experts and progressive knowledge transfer to efficiently train small-scale multimodal language models with limited computational resources while outperforming larger models.
paper_ai_keywords: Mixture of Experts (MoE) mimic distillation Kullback-Leibler (KL) divergence preference distillation Direct Preference Optimization (DPO) multimodal language models efficient training computational efficiency knowledge migration hallucination benchmarks