Law of Vision Representation in MLLMs
We present the "Law of Vision Representation" in multimodal large language
models (MLLMs). It reveals a strong correlation between the combination of
cross-modal alignment, correspondence in vision representation, and MLLM
performance. We quantify the two factors using the cross-modal Alignment and
Correspondence score (AC score). Through extensive experiments involving
thirteen different vision representation settings and evaluations across eight
benchmarks, we find that the AC score is linearly correlated to model
performance. By leveraging this relationship, we are able to identify and train
the optimal vision representation only, which does not require finetuning the
language model every time, resulting in a 99.7% reduction in computational
cost.
66d125133da2d178ceee1935
Shijia Yang
claimed_verified
2024-08-30T07:49:37.279Z
64374254a701a7e744c033a1
/avatars/a103d27fe32c520cb51191920a5b548a.svg
Shijia Yang
user
shijiay
66d125133da2d178ceee1936
Bohan Zhai
admin_assigned
2024-08-30T08:43:03.322Z
642fb0309b2484d7d857dbae
/avatars/7b10993e665198e80bd6627bf584ed71.svg
Bohan Zhai
user
Borise
66d125133da2d178ceee1937
Quanzeng You
admin_assigned
2024-08-30T08:43:11.747Z
642bb8c87f152f6e72b8d3cd
https://cdn-avatars.huggingface.co/v1/production/uploads/642bb8c87f152f6e72b8d3cd/MXxTssocydbabGZ6XFIic.png
Quanzeng You
user
Ye27
66d125133da2d178ceee1938
Jianbo Yuan
admin_assigned
2024-08-30T08:43:22.508Z
656d32d99ced9d5ff5846233
/avatars/e02f42e5a4f6ccfaa1aaf345eda47dea.svg
Yuan JianBo
user
WaterInSea
66d125133da2d178ceee1939
Hongxia Yang
66d125133da2d178ceee193a
Chenfeng Xu
claimed_verified
2024-08-30T07:18:01.667Z
654ca71d5255ee86711b52c5
/avatars/52bf00fd74c8db5643c4daa185c678e6.svg
Chenfeng Xu
user
chenfengx
Law of Vision Representation in MLLMs
We present the "Law of Vision Representation" in multimodal large language
models (MLLMs). It reveals a strong correlation between the combination of
cross-modal alignment, correspondence in vision representation, and MLLM
performance. We quantify the two factors using the cross-modal Alignment and
Correspondence score (AC score). Through extensive experiments involving
thirteen different vision representation settings and evaluations across eight
benchmarks, we find that the AC score is linearly correlated to model
performance. By leveraging this relationship, we are able to identify and train
the optimal vision representation only, which does not require finetuning the
language model every time, resulting in a 99.7% reduction in computational
cost.
Correlation between cross-modal alignment and vision representation improves performance in multimodal large language models, enabling identification and training of optimal vision representation with reduced computational cost.
cross-modal alignment vision representation MLLMs AC score multimodal large language models