Deciphering Cross-Modal Alignment in Large Vision-Language Models with
  Modality Integration Rate
We present the Modality Integration Rate (MIR), an effective, robust, and
generalized metric to indicate the multi-modal pre-training quality of Large
Vision Language Models (LVLMs). Large-scale pre-training plays a critical role
in building capable LVLMs, while evaluating its training quality without the
costly supervised fine-tuning stage is under-explored. Loss, perplexity, and
in-context evaluation results are commonly used pre-training metrics for Large
Language Models (LLMs), while we observed that these metrics are less
indicative when aligning a well-trained LLM with a new modality. Due to the
lack of proper metrics, the research of LVLMs in the critical pre-training
stage is hindered greatly, including the training data choice, efficient module
design, etc. In this paper, we propose evaluating the pre-training quality from
the inter-modal distribution distance perspective and present MIR, the Modality
Integration Rate, which is 1) Effective to represent the pre-training
quality and show a positive relation with the benchmark performance after
supervised fine-tuning. 2) Robust toward different training/evaluation
data. 3) Generalize across training configurations and architecture
choices. We conduct a series of pre-training experiments to explore the
effectiveness of MIR and observe satisfactory results that MIR is indicative
about training data selection, training strategy schedule, and model
architecture design to get better pre-training results. We hope MIR could be a
helpful metric for building capable LVLMs and inspire the following research
about modality alignment in different areas. Our code is at:
https://github.com/shikiw/Modality-Integration-Rate.
670748db49d08fabeb1ae5cd
Qidong Huang
admin_assigned
2024-10-10T08:37:24.039Z
656f1b21b075b63c90ba02ee
/avatars/d6856815ef06261394178161e4d511b4.svg
Huang Qidong
user
shikiw
670748db49d08fabeb1ae5ce
Xiaoyi Dong
670748db49d08fabeb1ae5cf
Pan Zhang
670748db49d08fabeb1ae5d0
Yuhang Zang
admin_assigned
2024-10-10T08:38:00.537Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
670748db49d08fabeb1ae5d1
Yuhang Cao
670748db49d08fabeb1ae5d2
Jiaqi Wang
claimed_verified
2024-12-11T09:21:53.724Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
670748db49d08fabeb1ae5d3
Dahua Lin
admin_assigned
2024-10-10T08:39:08.787Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
670748db49d08fabeb1ae5d4
Weiming Zhang
670748db49d08fabeb1ae5d5
Nenghai Yu
Deciphering Cross-Modal Alignment in Large Vision-Language Models with
  Modality Integration Rate
We present the Modality Integration Rate (MIR), an effective, robust, and
generalized metric to indicate the multi-modal pre-training quality of Large
Vision Language Models (LVLMs). Large-scale pre-training plays a critical role
in building capable LVLMs, while evaluating its training quality without the
costly supervised fine-tuning stage is under-explored. Loss, perplexity, and
in-context evaluation results are commonly used pre-training metrics for Large
Language Models (LLMs), while we observed that these metrics are less
indicative when aligning a well-trained LLM with a new modality. Due to the
lack of proper metrics, the research of LVLMs in the critical pre-training
stage is hindered greatly, including the training data choice, efficient module
design, etc. In this paper, we propose evaluating the pre-training quality from
the inter-modal distribution distance perspective and present MIR, the Modality
Integration Rate, which is 1) Effective to represent the pre-training
quality and show a positive relation with the benchmark performance after
supervised fine-tuning. 2) Robust toward different training/evaluation
data. 3) Generalize across training configurations and architecture
choices. We conduct a series of pre-training experiments to explore the
effectiveness of MIR and observe satisfactory results that MIR is indicative
about training data selection, training strategy schedule, and model
architecture design to get better pre-training results. We hope MIR could be a
helpful metric for building capable LVLMs and inspire the following research
about modality alignment in different areas. Our code is at:
https://github.com/shikiw/Modality-Integration-Rate.
MIR, a new metric for evaluating multi-modal pre-training in Large Vision Language Models, effectively correlates with benchmark performance, is robust across data, and generalizes well, aiding in data selection, training strategy, and architecture design.
MIR Modality Integration Rate Large Vision Language Models LVLMs Large Language Models LLMs pre-training quality in-context evaluation inter-modal distribution distance benchmark performance supervised fine-tuning training data selection training strategy schedule model architecture design