HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
  Large Language Models
Recent advancements indicate that scaling up Multimodal Large Language Models
(MLLMs) effectively enhances performance on downstream multimodal tasks. The
prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into
text-like tokens using a static vision-language mapper, thereby enabling
static LLMs to develop the capability to comprehend visual information
through visual instruction tuning. Although promising, the static tuning
strategy~The static tuning refers to the trained model with static
parameters. that shares the same parameters may constrain performance across
different downstream multimodal tasks. In light of this, we introduce
HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters,
in conjunction with a dynamic visual expert and language expert, respectively.
These experts are derived from HyperNetworks, which generates adaptive
parameter shifts through visual and language guidance, enabling dynamic
projector and LLM modeling in two-stage training.
  Our experiments demonstrate that our solution significantly surpasses LLaVA
on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and
LLaVA-Bench. ~Our project is available on the link
https://github.com/DCDmllm/HyperLLaVA.
65fbb0088027e00bcc6569fd
Wenqiao Zhang
claimed_verified
2024-03-22T08:38:11.046Z
65fc18edfb66882aba4d548e
/avatars/f70d47fe4aba98b5a5cd64f7e002dfd2.svg
wenqiao
user
wannature
65fbb0088027e00bcc6569fe
Tianwei Lin
65fbb0088027e00bcc6569ff
Jiang Liu
65fbb0088027e00bcc656a00
Fangxun Shu
claimed_verified
2024-09-10T12:26:08.744Z
6491b1b2c1741666238f8a0f
/avatars/9246c9ef06d80bd8628426375c95d4be.svg
JackShu
user
Shuhuhuhu
65fbb0088027e00bcc656a01
Haoyuan Li
65fbb0088027e00bcc656a02
Lei Zhang
65fbb0088027e00bcc656a03
He Wanggui
65fbb0088027e00bcc656a04
Hao Zhou
65fbb0088027e00bcc656a05
Zheqi Lv
65fbb0088027e00bcc656a06
Hao Jiang
65fbb0088027e00bcc656a07
Juncheng Li
admin_assigned
2024-03-21T10:56:47.489Z
63648c94f2984121733ec824
/avatars/93e3762d47b873d9dece4930cd96fa6b.svg
Billy
user
lijuncheng16
65fbb0088027e00bcc656a08
Siliang Tang
65fbb0088027e00bcc656a09
Yueting Zhuang
HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal
  Large Language Models
Recent advancements indicate that scaling up Multimodal Large Language Models
(MLLMs) effectively enhances performance on downstream multimodal tasks. The
prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into
text-like tokens using a static vision-language mapper, thereby enabling
static LLMs to develop the capability to comprehend visual information
through visual instruction tuning. Although promising, the static tuning
strategy~The static tuning refers to the trained model with static
parameters. that shares the same parameters may constrain performance across
different downstream multimodal tasks. In light of this, we introduce
HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters,
in conjunction with a dynamic visual expert and language expert, respectively.
These experts are derived from HyperNetworks, which generates adaptive
parameter shifts through visual and language guidance, enabling dynamic
projector and LLM modeling in two-stage training.
  Our experiments demonstrate that our solution significantly surpasses LLaVA
on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and
LLaVA-Bench. ~Our project is available on the link
https://github.com/DCDmllm/HyperLLaVA.
HyperLLaVA improves multimodal language model performance through adaptive tuning and dynamic experts derived from HyperNetworks.
multimodal large language models MLLMs LLaVA visual instruction tuning HyperLLaVA adaptive tuning projector LLM parameters dynamic visual expert dynamic language expert HyperNetworks two-stage training MME MMBench SEED-Bench LLaVA-Bench