title: γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large
    Language Models
summary: Despite the significant progress in multimodal large language models (MLLMs),
  their high computational cost remains a barrier to real-world deployment.
  Inspired by the mixture of depths (MoDs) in natural language processing, we aim
  to address this limitation from the perspective of ``activated tokens''. Our
  key insight is that if most tokens are redundant for the layer computation,
  then can be skipped directly via the MoD layer. However, directly converting
  the dense layers of MLLMs to MoD layers leads to substantial performance
  degradation. To address this issue, we propose an innovative MoD adaptation
  strategy for existing MLLMs called gamma-MoD. In gamma-MoD, a novel
  metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of
  attention maps (ARank). Through ARank, we can effectively identify which layer
  is redundant and should be replaced with the MoD layer. Based on ARank, we
  further propose two novel designs to maximize the computational sparsity of
  MLLM while maintaining its performance, namely shared vision-language router
  and masked routing learning. With these designs, more than 90% dense layers of
  the MLLM can be effectively converted to the MoD ones. To validate our method,
  we apply it to three popular MLLMs, and conduct extensive experiments on 9
  benchmark datasets. Experimental results not only validate the significant
  efficiency benefit of gamma-MoD to existing MLLMs but also confirm its
  generalization ability on various MLLMs. For example, with a minor performance
  drop, i.e., -1.5%, gamma-MoD can reduce the training and inference time of
  LLaVA-HR by 31.0% and 53.2%, respectively.
paper_authors._id: 6712229d0babf0ff2ee7d3a6
paper_authors.name: Yaxin Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T09:25:01.030Z
paper_authors._id: 653cb809b424289c5f384a02
paper_authors.avatarUrl: /avatars/a1565ab5ae51075c75d6857d64c426a8.svg
paper_authors.fullname: YaxinLuo
paper_authors.type: user
paper_authors.user: YaxinLuo
paper_authors._id: 6712229d0babf0ff2ee7d3a7
paper_authors.name: Gen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:03:57.293Z
paper_authors._id: 650aac7c23196fb2d86a0b37
paper_authors.avatarUrl: /avatars/418035a2e8f514118bc67d16ee41b6b0.svg
paper_authors.fullname: Gen Luo
paper_authors.type: user
paper_authors.user: favor123
paper_authors._id: 6712229d0babf0ff2ee7d3a8
paper_authors.name: Jiayi Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:03:51.931Z
paper_authors._id: 61c7cb3fc4157c1d283da869
paper_authors.avatarUrl: /avatars/8707bcb8b4e3725fe43115a88fe14be9.svg
paper_authors.fullname: Jiayi Ji
paper_authors.type: user
paper_authors.user: amy1254
paper_authors._id: 6712229d0babf0ff2ee7d3a9
paper_authors.name: Yiyi Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:03:46.428Z
paper_authors._id: 64e0d3b0c20c27fcc8cb0b86
paper_authors.avatarUrl: /avatars/eab7ee1f2d61d336c105a1d07c57df1c.svg
paper_authors.fullname: zhouyiyi
paper_authors.type: user
paper_authors.user: zhouyi1989
paper_authors._id: 6712229d0babf0ff2ee7d3aa
paper_authors.name: Xiaoshuai Sun
paper_authors._id: 6712229d0babf0ff2ee7d3ab
paper_authors.name: Zhiqiang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-18T16:03:19.045Z
paper_authors._id: 649d5a3f4e08d59d20014652
paper_authors.avatarUrl: /avatars/017eccd6aec02137c1beebfe4c951720.svg
paper_authors.fullname: Zhiqiang Shen
paper_authors.type: user
paper_authors.user: Jason0214
paper_authors._id: 6712229d0babf0ff2ee7d3ac
paper_authors.name: Rongrong Ji
paper_title: γ-MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large
    Language Models
paper_summary: Despite the significant progress in multimodal large language models (MLLMs),
  their high computational cost remains a barrier to real-world deployment.
  Inspired by the mixture of depths (MoDs) in natural language processing, we aim
  to address this limitation from the perspective of ``activated tokens''. Our
  key insight is that if most tokens are redundant for the layer computation,
  then can be skipped directly via the MoD layer. However, directly converting
  the dense layers of MLLMs to MoD layers leads to substantial performance
  degradation. To address this issue, we propose an innovative MoD adaptation
  strategy for existing MLLMs called gamma-MoD. In gamma-MoD, a novel
  metric is proposed to guide the deployment of MoDs in the MLLM, namely rank of
  attention maps (ARank). Through ARank, we can effectively identify which layer
  is redundant and should be replaced with the MoD layer. Based on ARank, we
  further propose two novel designs to maximize the computational sparsity of
  MLLM while maintaining its performance, namely shared vision-language router
  and masked routing learning. With these designs, more than 90% dense layers of
  the MLLM can be effectively converted to the MoD ones. To validate our method,
  we apply it to three popular MLLMs, and conduct extensive experiments on 9
  benchmark datasets. Experimental results not only validate the significant
  efficiency benefit of gamma-MoD to existing MLLMs but also confirm its
  generalization ability on various MLLMs. For example, with a minor performance
  drop, i.e., -1.5%, gamma-MoD can reduce the training and inference time of
  LLaVA-HR by 31.0% and 53.2%, respectively.
paper_ai_summary: The γ-MoD strategy, leveraging attention maps ranking and novel routing designs, efficiently reduces computational costs in multimodal large language models with minimal performance loss.
paper_ai_keywords: multimodal large language models mixture of depths MoD layers rank of attention maps ARank shared vision-language router masked routing learning