p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay
Despite the remarkable performance of multimodal large language models
(MLLMs) across diverse tasks, the substantial training and inference costs
impede their advancement. The majority of computation stems from the
overwhelming volume of vision tokens processed by the transformer decoder. In
this paper, we propose to build efficient MLLMs by leveraging the
Mixture-of-Depths (MoD) mechanism, where each transformer decoder layer selects
essential vision tokens to process while skipping redundant ones. However,
integrating MoD into MLLMs is non-trivial. To address the challenges of
training and inference stability as well as limited training data, we adapt the
MoD module with two novel designs: tanh-gated weight normalization (TanhNorm)
and symmetric token reweighting (STRing). Moreover, we observe that vision
tokens exhibit higher redundancy in deeper layer and thus design a progressive
ratio decay (PRD) strategy, which gradually reduces the token retention ratio
layer by layer, employing a shifted cosine schedule. This crucial design fully
unleashes the potential of MoD, significantly boosting the efficiency and
performance of our models. To validate the effectiveness of our approach, we
conduct extensive experiments with two baseline models across 14 benchmarks.
Our model, p-MoD, matches or even surpasses the performance of the baseline
models, with only 55.6% TFLOPs and 53.8% KV cache storage during inference, and
77.7% GPU hours during training.
67526e21ed8e5f5ec30abcbd
Jun Zhang
claimed_verified
2024-12-06T17:14:29.388Z
65372e922c6ef949b22c26d9
/avatars/f59adbdd576f731bb122c20fe8329391.svg
Jun Zhang
user
JungleGym
67526e21ed8e5f5ec30abcbe
Desen Meng
claimed_verified
2024-12-09T11:07:20.408Z
662a3abe6ffb912f2b3e6a54
/avatars/85782de6cd2dacd68efb0c8f488d9396.svg
dsmeng
user
dsmeng
67526e21ed8e5f5ec30abcbf
Ji Qi
67526e21ed8e5f5ec30abcc0
Zhenpeng Huang
claimed_verified
2025-04-16T09:42:22.818Z
62bec18e7e808565cc15610f
/avatars/78e8b98120a61bf90c43bd8c8ea8d375.svg
Huang Zhenpeng
user
hzp
67526e21ed8e5f5ec30abcc1
Tao Wu
67526e21ed8e5f5ec30abcc2
Limin Wang
p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay
Despite the remarkable performance of multimodal large language models
(MLLMs) across diverse tasks, the substantial training and inference costs
impede their advancement. The majority of computation stems from the
overwhelming volume of vision tokens processed by the transformer decoder. In
this paper, we propose to build efficient MLLMs by leveraging the
Mixture-of-Depths (MoD) mechanism, where each transformer decoder layer selects
essential vision tokens to process while skipping redundant ones. However,
integrating MoD into MLLMs is non-trivial. To address the challenges of
training and inference stability as well as limited training data, we adapt the
MoD module with two novel designs: tanh-gated weight normalization (TanhNorm)
and symmetric token reweighting (STRing). Moreover, we observe that vision
tokens exhibit higher redundancy in deeper layer and thus design a progressive
ratio decay (PRD) strategy, which gradually reduces the token retention ratio
layer by layer, employing a shifted cosine schedule. This crucial design fully
unleashes the potential of MoD, significantly boosting the efficiency and
performance of our models. To validate the effectiveness of our approach, we
conduct extensive experiments with two baseline models across 14 benchmarks.
Our model, p-MoD, matches or even surpasses the performance of the baseline
models, with only 55.6% TFLOPs and 53.8% KV cache storage during inference, and
77.7% GPU hours during training.
The paper presents a Mixture-of-Depths (MoD) mechanism with novel adaptations to enhance efficiency and performance in multimodal large language models (MLLMs) by selectively processing vision tokens.
Mixture-of-Depths transformer decoder vision tokens tanh-gated weight normalization symmetric token reweighting progressive ratio decay shifted cosine schedule p-MoD TFLOPs KV cache storage