title: AlignGPT: Multi-modal Large Language Models with Adaptive Alignment
    Capability
summary: Multimodal Large Language Models (MLLMs) are widely regarded as crucial in
  the exploration of Artificial General Intelligence (AGI). The core of MLLMs
  lies in their capability to achieve cross-modal alignment. To attain this goal,
  current MLLMs typically follow a two-phase training paradigm: the pre-training
  phase and the instruction-tuning phase. Despite their success, there are
  shortcomings in the modeling of alignment capabilities within these models.
  Firstly, during the pre-training phase, the model usually assumes that all
  image-text pairs are uniformly aligned, but in fact the degree of alignment
  between different image-text pairs is inconsistent. Secondly, the instructions
  currently used for finetuning incorporate a variety of tasks, different tasks's
  instructions usually require different levels of alignment capabilities, but
  previous MLLMs overlook these differentiated alignment needs. To tackle these
  issues, we propose a new multimodal large language model AlignGPT. In the
  pre-training stage, instead of treating all image-text pairs equally, we assign
  different levels of alignment capabilities to different image-text pairs. Then,
  in the instruction-tuning phase, we adaptively combine these different levels
  of alignment capabilities to meet the dynamic alignment needs of different
  instructions. Extensive experimental results show that our model achieves
  competitive performance on 12 benchmarks.
paper_authors._id: 66501fe060fc035edd879d5a
paper_authors.name: Fei Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-24T15:44:38.309Z
paper_authors._id: 643660b5fae28700514a138d
paper_authors.avatarUrl: /avatars/a48bddbfdcedc0dd2c2836202b3c8207.svg
paper_authors.fullname: zhaofei
paper_authors.type: user
paper_authors.user: nlpzhaof
paper_authors._id: 66501fe060fc035edd879d5b
paper_authors.name: Taotian Pang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:57:30.613Z
paper_authors._id: 64b10448d47ab770167865e4
paper_authors.avatarUrl: /avatars/d56a3210f1460c4c46fe782eee9cb8c0.svg
paper_authors.fullname: Taotian Pang
paper_authors.type: user
paper_authors.user: TJTP
paper_authors._id: 66501fe060fc035edd879d5c
paper_authors.name: Chunhui Li
paper_authors._id: 66501fe060fc035edd879d5d
paper_authors.name: Zhen Wu
paper_authors._id: 66501fe060fc035edd879d5e
paper_authors.name: Junjie Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:59:16.605Z
paper_authors._id: 639598cb2caa32c0d334a545
paper_authors.avatarUrl: /avatars/2751ce681232b0b4666f87fd3f20aebd.svg
paper_authors.fullname: Junjie Guo
paper_authors.type: user
paper_authors.user: TealOrange
paper_authors._id: 66501fe060fc035edd879d5f
paper_authors.name: Shangyu Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-24T11:59:22.829Z
paper_authors._id: 64bc8f584d2052b1aa062b91
paper_authors.avatarUrl: /avatars/c63f3e4787f463d36bcdf8c15d8214c1.svg
paper_authors.fullname: Shangyu Xing
paper_authors.type: user
paper_authors.user: starreeze
paper_authors._id: 66501fe060fc035edd879d60
paper_authors.name: Xinyu Dai
paper_title: AlignGPT: Multi-modal Large Language Models with Adaptive Alignment
    Capability
paper_summary: Multimodal Large Language Models (MLLMs) are widely regarded as crucial in
  the exploration of Artificial General Intelligence (AGI). The core of MLLMs
  lies in their capability to achieve cross-modal alignment. To attain this goal,
  current MLLMs typically follow a two-phase training paradigm: the pre-training
  phase and the instruction-tuning phase. Despite their success, there are
  shortcomings in the modeling of alignment capabilities within these models.
  Firstly, during the pre-training phase, the model usually assumes that all
  image-text pairs are uniformly aligned, but in fact the degree of alignment
  between different image-text pairs is inconsistent. Secondly, the instructions
  currently used for finetuning incorporate a variety of tasks, different tasks's
  instructions usually require different levels of alignment capabilities, but
  previous MLLMs overlook these differentiated alignment needs. To tackle these
  issues, we propose a new multimodal large language model AlignGPT. In the
  pre-training stage, instead of treating all image-text pairs equally, we assign
  different levels of alignment capabilities to different image-text pairs. Then,
  in the instruction-tuning phase, we adaptively combine these different levels
  of alignment capabilities to meet the dynamic alignment needs of different
  instructions. Extensive experimental results show that our model achieves
  competitive performance on 12 benchmarks.
paper_ai_summary: AlignGPT addresses alignment inconsistencies in multimodal large language models by dynamically assigning and combining varying alignment capabilities during pre-training and instruction tuning phases.
paper_ai_keywords: Multimodal Large Language Models AGI cross-modal alignment pre-training phase instruction-tuning phase image-text pairs AlignGPT