γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large
  Language Models
Despite the significant progress in multimodal large language models (MLLMs),
their high computational cost remains a barrier to real-world deployment.
Inspired by the mixture of depths (MoDs) in natural language processing, we aim
to address this limitation from the perspective of ``activated tokens''. Our
key insight is that if most tokens are redundant for the layer computation,
then can be skipped directly via the MoD layer. However, directly converting
the dense layers of MLLMs to MoD layers leads to substantial performance
degradation. To address this issue, we propose an innovative MoD adaptation
strategy for existing MLLMs called gamma-MoD. In gamma-MoD, a novel
metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of
attention maps (ARank). Through ARank, we can effectively identify which layer
is redundant and should be replaced with the MoD layer. Based on ARank, we
further propose two novel designs to maximize the computational sparsity of
MLLM while maintaining its performance, namely shared vision-language router
and masked routing learning. With these designs, more than 90% dense layers of
the MLLM can be effectively converted to the MoD ones. To validate our method,
we apply it to three popular MLLMs, and conduct extensive experiments on 9
benchmark datasets. Experimental results not only validate the significant
efficiency benefit of gamma-MoD to existing MLLMs but also confirm its
generalization ability on various MLLMs. For example, with a minor performance
drop, i.e., -1.5%, gamma-MoD can reduce the training and inference time of
LLaVA-HR by 31.0% and 53.2%, respectively.
6712229d0babf0ff2ee7d3a6
Yaxin Luo
claimed_verified
2024-10-18T09:25:01.030Z
653cb809b424289c5f384a02
/avatars/a1565ab5ae51075c75d6857d64c426a8.svg
YaxinLuo
user
YaxinLuo
6712229d0babf0ff2ee7d3a7
Gen Luo
admin_assigned
2024-10-18T16:03:57.293Z
650aac7c23196fb2d86a0b37
/avatars/418035a2e8f514118bc67d16ee41b6b0.svg
Gen Luo
user
favor123
6712229d0babf0ff2ee7d3a8
Jiayi Ji
admin_assigned
2024-10-18T16:03:51.931Z
61c7cb3fc4157c1d283da869
/avatars/8707bcb8b4e3725fe43115a88fe14be9.svg
Jiayi Ji
user
amy1254
6712229d0babf0ff2ee7d3a9
Yiyi Zhou
admin_assigned
2024-10-18T16:03:46.428Z
64e0d3b0c20c27fcc8cb0b86
/avatars/eab7ee1f2d61d336c105a1d07c57df1c.svg
zhouyiyi
user
zhouyi1989
6712229d0babf0ff2ee7d3aa
Xiaoshuai Sun
6712229d0babf0ff2ee7d3ab
Zhiqiang Shen
admin_assigned
2024-10-18T16:03:19.045Z
649d5a3f4e08d59d20014652
/avatars/017eccd6aec02137c1beebfe4c951720.svg
Zhiqiang Shen
user
Jason0214
6712229d0babf0ff2ee7d3ac
Rongrong Ji
γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large
  Language Models
Despite the significant progress in multimodal large language models (MLLMs),
their high computational cost remains a barrier to real-world deployment.
Inspired by the mixture of depths (MoDs) in natural language processing, we aim
to address this limitation from the perspective of ``activated tokens''. Our
key insight is that if most tokens are redundant for the layer computation,
then can be skipped directly via the MoD layer. However, directly converting
the dense layers of MLLMs to MoD layers leads to substantial performance
degradation. To address this issue, we propose an innovative MoD adaptation
strategy for existing MLLMs called gamma-MoD. In gamma-MoD, a novel
metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of
attention maps (ARank). Through ARank, we can effectively identify which layer
is redundant and should be replaced with the MoD layer. Based on ARank, we
further propose two novel designs to maximize the computational sparsity of
MLLM while maintaining its performance, namely shared vision-language router
and masked routing learning. With these designs, more than 90% dense layers of
the MLLM can be effectively converted to the MoD ones. To validate our method,
we apply it to three popular MLLMs, and conduct extensive experiments on 9
benchmark datasets. Experimental results not only validate the significant
efficiency benefit of gamma-MoD to existing MLLMs but also confirm its
generalization ability on various MLLMs. For example, with a minor performance
drop, i.e., -1.5%, gamma-MoD can reduce the training and inference time of
LLaVA-HR by 31.0% and 53.2%, respectively.
The γ-MoD strategy, leveraging attention maps ranking and novel routing designs, efficiently reduces computational costs in multimodal large language models with minimal performance loss.
multimodal large language models mixture of depths MoD layers rank of attention maps ARank shared vision-language router masked routing learning