title: Deciphering Cross-Modal Alignment in Large Vision-Language Models with
    Modality Integration Rate
summary: We present the Modality Integration Rate (MIR), an effective, robust, and
  generalized metric to indicate the multi-modal pre-training quality of Large
  Vision Language Models (LVLMs). Large-scale pre-training plays a critical role
  in building capable LVLMs, while evaluating its training quality without the
  costly supervised fine-tuning stage is under-explored. Loss, perplexity, and
  in-context evaluation results are commonly used pre-training metrics for Large
  Language Models (LLMs), while we observed that these metrics are less
  indicative when aligning a well-trained LLM with a new modality. Due to the
  lack of proper metrics, the research of LVLMs in the critical pre-training
  stage is hindered greatly, including the training data choice, efficient module
  design, etc. In this paper, we propose evaluating the pre-training quality from
  the inter-modal distribution distance perspective and present MIR, the Modality
  Integration Rate, which is 1) Effective to represent the pre-training
  quality and show a positive relation with the benchmark performance after
  supervised fine-tuning. 2) Robust toward different training/evaluation
  data. 3) Generalize across training configurations and architecture
  choices. We conduct a series of pre-training experiments to explore the
  effectiveness of MIR and observe satisfactory results that MIR is indicative
  about training data selection, training strategy schedule, and model
  architecture design to get better pre-training results. We hope MIR could be a
  helpful metric for building capable LVLMs and inspire the following research
  about modality alignment in different areas. Our code is at:
  https://github.com/shikiw/Modality-Integration-Rate.
paper_authors._id: 670748db49d08fabeb1ae5cd
paper_authors.name: Qidong Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:37:24.039Z
paper_authors._id: 656f1b21b075b63c90ba02ee
paper_authors.avatarUrl: /avatars/d6856815ef06261394178161e4d511b4.svg
paper_authors.fullname: Huang Qidong
paper_authors.type: user
paper_authors.user: shikiw
paper_authors._id: 670748db49d08fabeb1ae5ce
paper_authors.name: Xiaoyi Dong
paper_authors._id: 670748db49d08fabeb1ae5cf
paper_authors.name: Pan Zhang
paper_authors._id: 670748db49d08fabeb1ae5d0
paper_authors.name: Yuhang Zang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:38:00.537Z
paper_authors._id: 63859cf3b2906edaf83af9f0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
paper_authors.fullname: Yuhang Zang
paper_authors.type: user
paper_authors.user: yuhangzang
paper_authors._id: 670748db49d08fabeb1ae5d1
paper_authors.name: Yuhang Cao
paper_authors._id: 670748db49d08fabeb1ae5d2
paper_authors.name: Jiaqi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-11T09:21:53.724Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_authors._id: 670748db49d08fabeb1ae5d3
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T08:39:08.787Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 670748db49d08fabeb1ae5d4
paper_authors.name: Weiming Zhang
paper_authors._id: 670748db49d08fabeb1ae5d5
paper_authors.name: Nenghai Yu
paper_title: Deciphering Cross-Modal Alignment in Large Vision-Language Models with
    Modality Integration Rate
paper_summary: We present the Modality Integration Rate (MIR), an effective, robust, and
  generalized metric to indicate the multi-modal pre-training quality of Large
  Vision Language Models (LVLMs). Large-scale pre-training plays a critical role
  in building capable LVLMs, while evaluating its training quality without the
  costly supervised fine-tuning stage is under-explored. Loss, perplexity, and
  in-context evaluation results are commonly used pre-training metrics for Large
  Language Models (LLMs), while we observed that these metrics are less
  indicative when aligning a well-trained LLM with a new modality. Due to the
  lack of proper metrics, the research of LVLMs in the critical pre-training
  stage is hindered greatly, including the training data choice, efficient module
  design, etc. In this paper, we propose evaluating the pre-training quality from
  the inter-modal distribution distance perspective and present MIR, the Modality
  Integration Rate, which is 1) Effective to represent the pre-training
  quality and show a positive relation with the benchmark performance after
  supervised fine-tuning. 2) Robust toward different training/evaluation
  data. 3) Generalize across training configurations and architecture
  choices. We conduct a series of pre-training experiments to explore the
  effectiveness of MIR and observe satisfactory results that MIR is indicative
  about training data selection, training strategy schedule, and model
  architecture design to get better pre-training results. We hope MIR could be a
  helpful metric for building capable LVLMs and inspire the following research
  about modality alignment in different areas. Our code is at:
  https://github.com/shikiw/Modality-Integration-Rate.
paper_ai_summary: MIR, a new metric for evaluating multi-modal pre-training in Large Vision Language Models, effectively correlates with benchmark performance, is robust across data, and generalizes well, aiding in data selection, training strategy, and architecture design.
paper_ai_keywords: MIR Modality Integration Rate Large Vision Language Models LVLMs Large Language Models LLMs pre-training quality in-context evaluation inter-modal distribution distance benchmark performance supervised fine-tuning training data selection training strategy schedule model architecture design