title: HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
    Large Language Models
summary: Recent advancements indicate that scaling up Multimodal Large Language Models
  (MLLMs) effectively enhances performance on downstream multimodal tasks. The
  prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into
  text-like tokens using a static vision-language mapper, thereby enabling
  static LLMs to develop the capability to comprehend visual information
  through visual instruction tuning. Although promising, the static tuning
  strategy~The static tuning refers to the trained model with static
  parameters. that shares the same parameters may constrain performance across
  different downstream multimodal tasks. In light of this, we introduce
  HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters,
  in conjunction with a dynamic visual expert and language expert, respectively.
  These experts are derived from HyperNetworks, which generates adaptive
  parameter shifts through visual and language guidance, enabling dynamic
  projector and LLM modeling in two-stage training.
    Our experiments demonstrate that our solution significantly surpasses LLaVA
  on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and
  LLaVA-Bench. ~Our project is available on the link
  https://github.com/DCDmllm/HyperLLaVA.
paper_authors._id: 65fbb0088027e00bcc6569fd
paper_authors.name: Wenqiao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:38:11.046Z
paper_authors._id: 65fc18edfb66882aba4d548e
paper_authors.avatarUrl: /avatars/f70d47fe4aba98b5a5cd64f7e002dfd2.svg
paper_authors.fullname: wenqiao
paper_authors.type: user
paper_authors.user: wannature
paper_authors._id: 65fbb0088027e00bcc6569fe
paper_authors.name: Tianwei Lin
paper_authors._id: 65fbb0088027e00bcc6569ff
paper_authors.name: Jiang Liu
paper_authors._id: 65fbb0088027e00bcc656a00
paper_authors.name: Fangxun Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-10T12:26:08.744Z
paper_authors._id: 6491b1b2c1741666238f8a0f
paper_authors.avatarUrl: /avatars/9246c9ef06d80bd8628426375c95d4be.svg
paper_authors.fullname: JackShu
paper_authors.type: user
paper_authors.user: Shuhuhuhu
paper_authors._id: 65fbb0088027e00bcc656a01
paper_authors.name: Haoyuan Li
paper_authors._id: 65fbb0088027e00bcc656a02
paper_authors.name: Lei Zhang
paper_authors._id: 65fbb0088027e00bcc656a03
paper_authors.name: He Wanggui
paper_authors._id: 65fbb0088027e00bcc656a04
paper_authors.name: Hao Zhou
paper_authors._id: 65fbb0088027e00bcc656a05
paper_authors.name: Zheqi Lv
paper_authors._id: 65fbb0088027e00bcc656a06
paper_authors.name: Hao Jiang
paper_authors._id: 65fbb0088027e00bcc656a07
paper_authors.name: Juncheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-21T10:56:47.489Z
paper_authors._id: 63648c94f2984121733ec824
paper_authors.avatarUrl: /avatars/93e3762d47b873d9dece4930cd96fa6b.svg
paper_authors.fullname: Billy
paper_authors.type: user
paper_authors.user: lijuncheng16
paper_authors._id: 65fbb0088027e00bcc656a08
paper_authors.name: Siliang Tang
paper_authors._id: 65fbb0088027e00bcc656a09
paper_authors.name: Yueting Zhuang
paper_title: HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
    Large Language Models
paper_summary: Recent advancements indicate that scaling up Multimodal Large Language Models
  (MLLMs) effectively enhances performance on downstream multimodal tasks. The
  prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into
  text-like tokens using a static vision-language mapper, thereby enabling
  static LLMs to develop the capability to comprehend visual information
  through visual instruction tuning. Although promising, the static tuning
  strategy~The static tuning refers to the trained model with static
  parameters. that shares the same parameters may constrain performance across
  different downstream multimodal tasks. In light of this, we introduce
  HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters,
  in conjunction with a dynamic visual expert and language expert, respectively.
  These experts are derived from HyperNetworks, which generates adaptive
  parameter shifts through visual and language guidance, enabling dynamic
  projector and LLM modeling in two-stage training.
    Our experiments demonstrate that our solution significantly surpasses LLaVA
  on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and
  LLaVA-Bench. ~Our project is available on the link
  https://github.com/DCDmllm/HyperLLaVA.
paper_ai_summary: HyperLLaVA improves multimodal language model performance through adaptive tuning and dynamic experts derived from HyperNetworks.
paper_ai_keywords: multimodal large language models MLLMs LLaVA visual instruction tuning HyperLLaVA adaptive tuning projector LLM parameters dynamic visual expert dynamic language expert HyperNetworks two-stage training MME MMBench SEED-Bench LLaVA-Bench