title: p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay
summary: Despite the remarkable performance of multimodal large language models
  (MLLMs) across diverse tasks, the substantial training and inference costs
  impede their advancement. The majority of computation stems from the
  overwhelming volume of vision tokens processed by the transformer decoder. In
  this paper, we propose to build efficient MLLMs by leveraging the
  Mixture-of-Depths (MoD) mechanism, where each transformer decoder layer selects
  essential vision tokens to process while skipping redundant ones. However,
  integrating MoD into MLLMs is non-trivial. To address the challenges of
  training and inference stability as well as limited training data, we adapt the
  MoD module with two novel designs: tanh-gated weight normalization (TanhNorm)
  and symmetric token reweighting (STRing). Moreover, we observe that vision
  tokens exhibit higher redundancy in deeper layer and thus design a progressive
  ratio decay (PRD) strategy, which gradually reduces the token retention ratio
  layer by layer, employing a shifted cosine schedule. This crucial design fully
  unleashes the potential of MoD, significantly boosting the efficiency and
  performance of our models. To validate the effectiveness of our approach, we
  conduct extensive experiments with two baseline models across 14 benchmarks.
  Our model, p-MoD, matches or even surpasses the performance of the baseline
  models, with only 55.6% TFLOPs and 53.8% KV cache storage during inference, and
  77.7% GPU hours during training.
paper_authors._id: 67526e21ed8e5f5ec30abcbd
paper_authors.name: Jun Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T17:14:29.388Z
paper_authors._id: 65372e922c6ef949b22c26d9
paper_authors.avatarUrl: /avatars/f59adbdd576f731bb122c20fe8329391.svg
paper_authors.fullname: Jun Zhang
paper_authors.type: user
paper_authors.user: JungleGym
paper_authors._id: 67526e21ed8e5f5ec30abcbe
paper_authors.name: Desen Meng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-09T11:07:20.408Z
paper_authors._id: 662a3abe6ffb912f2b3e6a54
paper_authors.avatarUrl: /avatars/85782de6cd2dacd68efb0c8f488d9396.svg
paper_authors.fullname: dsmeng
paper_authors.type: user
paper_authors.user: dsmeng
paper_authors._id: 67526e21ed8e5f5ec30abcbf
paper_authors.name: Ji Qi
paper_authors._id: 67526e21ed8e5f5ec30abcc0
paper_authors.name: Zhenpeng Huang
paper_authors._id: 67526e21ed8e5f5ec30abcc1
paper_authors.name: Tao Wu
paper_authors._id: 67526e21ed8e5f5ec30abcc2
paper_authors.name: Limin Wang
paper_title: p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay
paper_summary: Despite the remarkable performance of multimodal large language models
  (MLLMs) across diverse tasks, the substantial training and inference costs
  impede their advancement. The majority of computation stems from the
  overwhelming volume of vision tokens processed by the transformer decoder. In
  this paper, we propose to build efficient MLLMs by leveraging the
  Mixture-of-Depths (MoD) mechanism, where each transformer decoder layer selects
  essential vision tokens to process while skipping redundant ones. However,
  integrating MoD into MLLMs is non-trivial. To address the challenges of
  training and inference stability as well as limited training data, we adapt the
  MoD module with two novel designs: tanh-gated weight normalization (TanhNorm)
  and symmetric token reweighting (STRing). Moreover, we observe that vision
  tokens exhibit higher redundancy in deeper layer and thus design a progressive
  ratio decay (PRD) strategy, which gradually reduces the token retention ratio
  layer by layer, employing a shifted cosine schedule. This crucial design fully
  unleashes the potential of MoD, significantly boosting the efficiency and
  performance of our models. To validate the effectiveness of our approach, we
  conduct extensive experiments with two baseline models across 14 benchmarks.
  Our model, p-MoD, matches or even surpasses the performance of the baseline
  models, with only 55.6% TFLOPs and 53.8% KV cache storage during inference, and
  77.7% GPU hours during training.
paper_ai_keywords: Mixture-of-Depths (MoD) transformer decoder vision tokens tanh-gated weight normalization (TanhNorm) symmetric token reweighting (STRing) progressive ratio decay (PRD) shifted cosine schedule p-MoD