title: Law of Vision Representation in MLLMs
summary: We present the "Law of Vision Representation" in multimodal large language
  models (MLLMs). It reveals a strong correlation between the combination of
  cross-modal alignment, correspondence in vision representation, and MLLM
  performance. We quantify the two factors using the cross-modal Alignment and
  Correspondence score (AC score). Through extensive experiments involving
  thirteen different vision representation settings and evaluations across eight
  benchmarks, we find that the AC score is linearly correlated to model
  performance. By leveraging this relationship, we are able to identify and train
  the optimal vision representation only, which does not require finetuning the
  language model every time, resulting in a 99.7% reduction in computational
  cost.
paper_authors._id: 66d125133da2d178ceee1935
paper_authors.name: Shijia Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T07:49:37.279Z
paper_authors._id: 64374254a701a7e744c033a1
paper_authors.avatarUrl: /avatars/a103d27fe32c520cb51191920a5b548a.svg
paper_authors.fullname: Shijia Yang
paper_authors.type: user
paper_authors.user: shijiay
paper_authors._id: 66d125133da2d178ceee1936
paper_authors.name: Bohan Zhai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:43:03.322Z
paper_authors._id: 642fb0309b2484d7d857dbae
paper_authors.avatarUrl: /avatars/7b10993e665198e80bd6627bf584ed71.svg
paper_authors.fullname: Bohan Zhai
paper_authors.type: user
paper_authors.user: Borise
paper_authors._id: 66d125133da2d178ceee1937
paper_authors.name: Quanzeng You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:43:11.747Z
paper_authors._id: 642bb8c87f152f6e72b8d3cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642bb8c87f152f6e72b8d3cd/MXxTssocydbabGZ6XFIic.png
paper_authors.fullname: Quanzeng You
paper_authors.type: user
paper_authors.user: Ye27
paper_authors._id: 66d125133da2d178ceee1938
paper_authors.name: Jianbo Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:43:22.508Z
paper_authors._id: 656d32d99ced9d5ff5846233
paper_authors.avatarUrl: /avatars/e02f42e5a4f6ccfaa1aaf345eda47dea.svg
paper_authors.fullname: Yuan JianBo
paper_authors.type: user
paper_authors.user: WaterInSea
paper_authors._id: 66d125133da2d178ceee1939
paper_authors.name: Hongxia Yang
paper_authors._id: 66d125133da2d178ceee193a
paper_authors.name: Chenfeng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-30T07:18:01.667Z
paper_authors._id: 654ca71d5255ee86711b52c5
paper_authors.avatarUrl: /avatars/52bf00fd74c8db5643c4daa185c678e6.svg
paper_authors.fullname: Chenfeng Xu
paper_authors.type: user
paper_authors.user: chenfengx
paper_title: Law of Vision Representation in MLLMs
paper_summary: We present the "Law of Vision Representation" in multimodal large language
  models (MLLMs). It reveals a strong correlation between the combination of
  cross-modal alignment, correspondence in vision representation, and MLLM
  performance. We quantify the two factors using the cross-modal Alignment and
  Correspondence score (AC score). Through extensive experiments involving
  thirteen different vision representation settings and evaluations across eight
  benchmarks, we find that the AC score is linearly correlated to model
  performance. By leveraging this relationship, we are able to identify and train
  the optimal vision representation only, which does not require finetuning the
  language model every time, resulting in a 99.7% reduction in computational
  cost.
paper_ai_keywords: cross-modal alignment correspondence vision representation multimodal large language models AC score cross-modal Alignment and Correspondence score