LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
We introduce LLaVA-MoD, a novel framework designed to enable the efficient
training of small-scale Multimodal Language Models (s-MLLM) by distilling
knowledge from large-scale MLLM (l-MLLM). Our approach tackles two fundamental
challenges in MLLM distillation. First, we optimize the network structure of
s-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the
language model, striking a balance between computational efficiency and model
expressiveness. Second, we propose a progressive knowledge transfer strategy to
ensure comprehensive knowledge migration. This strategy begins with mimic
distillation, where we minimize the Kullback-Leibler (KL) divergence between
output distributions to enable the student model to emulate the teacher
network's understanding. Following this, we introduce preference distillation
via Direct Preference Optimization (DPO), where the key lies in treating l-MLLM
as the reference model. During this phase, the s-MLLM's ability to discriminate
between superior and inferior examples is significantly enhanced beyond l-MLLM,
leading to a better student that surpasses its teacher, particularly in
hallucination benchmarks. Extensive experiments demonstrate that LLaVA-MoD
outperforms existing models across various multimodal benchmarks while
maintaining a minimal number of activated parameters and low computational
costs. Remarkably, LLaVA-MoD, with only 2B activated parameters, surpasses
Qwen-VL-Chat-7B by an average of 8.8% across benchmarks, using merely 0.3% of
the training data and 23% trainable parameters. These results underscore
LLaVA-MoD's ability to effectively distill comprehensive knowledge from its
teacher model, paving the way for the development of more efficient MLLMs. The
code will be available on: https://github.com/shufangxun/LLaVA-MoD.
66cfdd7c39ce70f23b715ec2
Fangxun Shu
claimed_verified
2024-09-10T12:26:05.343Z
6491b1b2c1741666238f8a0f
/avatars/9246c9ef06d80bd8628426375c95d4be.svg
JackShu
user
Shuhuhuhu
66cfdd7c39ce70f23b715ec3
Yue Liao
admin_assigned
2024-08-29T08:11:04.820Z
666a5cd9148d701ff8340e8c
/avatars/8536b92af349e7b5f22d6b5bd67f77a1.svg
Yue Liao
user
yliao
66cfdd7c39ce70f23b715ec4
Le Zhuo
admin_assigned
2024-08-29T08:11:10.659Z
6358a167f56b03ec9147074d
/avatars/e54ea7bf0c240cf76d538296efb3976c.svg
Le Zhuo
user
JackyZhuo
66cfdd7c39ce70f23b715ec5
Chenning Xu
admin_assigned
2024-08-29T08:11:16.536Z
645c8fe95ebf379fd6d625e4
/avatars/728d039b9512d404a9048714ec74008b.svg
chenning xu
user
cnxu
66cfdd7c39ce70f23b715ec6
Guanghao Zhang
admin_assigned
2024-08-29T08:11:22.558Z
6662841168009120af048273
/avatars/dc6e5628d631998874f0b76d59eec051.svg
Guanghao Zhang
user
zghao
66cfdd7c39ce70f23b715ec7
Haonan Shi
66cfdd7c39ce70f23b715ec8
Long Chen
66cfdd7c39ce70f23b715ec9
Tao Zhong
66cfdd7c39ce70f23b715eca
Wanggui He
66cfdd7c39ce70f23b715ecb
Siming Fu
admin_assigned
2024-08-29T08:57:48.589Z
6485dd6d07a2c1915060f603
/avatars/8594d647359a7d19ab29b8ec91d1444e.svg
fu
user
simingfu
66cfdd7c39ce70f23b715ecc
Haoyuan Li
66cfdd7c39ce70f23b715ecd
Bolin Li
66cfdd7c39ce70f23b715ece
Zhelun Yu
66cfdd7c39ce70f23b715ecf
Si Liu
66cfdd7c39ce70f23b715ed0
Hongsheng Li
admin_assigned
2024-08-29T08:12:02.304Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
66cfdd7c39ce70f23b715ed1
Hao Jiang
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation
We introduce LLaVA-MoD, a novel framework designed to enable the efficient
training of small-scale Multimodal Language Models (s-MLLM) by distilling
knowledge from large-scale MLLM (l-MLLM). Our approach tackles two fundamental
challenges in MLLM distillation. First, we optimize the network structure of
s-MLLM by integrating a sparse Mixture of Experts (MoE) architecture into the
language model, striking a balance between computational efficiency and model
expressiveness. Second, we propose a progressive knowledge transfer strategy to
ensure comprehensive knowledge migration. This strategy begins with mimic
distillation, where we minimize the Kullback-Leibler (KL) divergence between
output distributions to enable the student model to emulate the teacher
network's understanding. Following this, we introduce preference distillation
via Direct Preference Optimization (DPO), where the key lies in treating l-MLLM
as the reference model. During this phase, the s-MLLM's ability to discriminate
between superior and inferior examples is significantly enhanced beyond l-MLLM,
leading to a better student that surpasses its teacher, particularly in
hallucination benchmarks. Extensive experiments demonstrate that LLaVA-MoD
outperforms existing models across various multimodal benchmarks while
maintaining a minimal number of activated parameters and low computational
costs. Remarkably, LLaVA-MoD, with only 2B activated parameters, surpasses
Qwen-VL-Chat-7B by an average of 8.8% across benchmarks, using merely 0.3% of
the training data and 23% trainable parameters. These results underscore
LLaVA-MoD's ability to effectively distill comprehensive knowledge from its
teacher model, paving the way for the development of more efficient MLLMs. The
code will be available on: https://github.com/shufangxun/LLaVA-MoD.
LLaVA-MoD employs a sparse Mixture of Experts and progressive knowledge transfer to efficiently train small-scale multimodal language models with limited computational resources while outperforming larger models.
Mixture of Experts (MoE) mimic distillation Kullback-Leibler (KL) divergence preference distillation Direct Preference Optimization (DPO) multimodal language models efficient training computational efficiency knowledge migration hallucination benchmarks