Parrot: Multilingual Visual Instruction Tuning
The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V
has marked a significant step towards artificial general intelligence. Existing
methods mainly focus on aligning vision encoders with LLMs through supervised
fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs'
inherent ability to react to multiple languages progressively deteriorate as
the training process evolves. We empirically find that the imbalanced SFT
datasets, primarily composed of English-centric image-text pairs, lead to
significantly reduced performance in non-English languages. This is due to the
failure of aligning the vision encoder and LLM with multilingual tokens during
the SFT process. In this paper, we introduce Parrot, a novel method that
utilizes textual guidance to drive visual token alignment at the language
level. Parrot makes the visual tokens condition on diverse language inputs and
uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens.
Specifically, to enhance non-English visual tokens alignment, we compute the
cross-attention using the initial visual features and textual embeddings, the
result of which is then fed into the MoE router to select the most relevant
experts. The selected experts subsequently convert the initial visual tokens
into language-specific visual tokens. Moreover, considering the current lack of
benchmarks for evaluating multilingual capabilities within the field, we
collect and make available a Massive Multilingual Multimodal Benchmark which
includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our
method not only demonstrates state-of-the-art performance on multilingual
MMBench and MMMB, but also excels across a broad range of multimodal tasks.
Both the source code and the training dataset of Parrot will be made publicly
available.
665fc9ffb99c631f4f55d34b
Hai-Long Sun
claimed_verified
2024-06-05T07:30:37.957Z
6623975c728f756224d4b768
https://cdn-avatars.huggingface.co/v1/production/uploads/6623975c728f756224d4b768/US9n0k45Y1TF6WqOhgBhZ.jpeg
Allen Sun
user
Allen8
665fc9ffb99c631f4f55d34c
Da-Wei Zhou
admin_assigned
2024-06-06T08:26:57.211Z
64254ff8120a3ed323314e0c
/avatars/134325256b9073a96c6d7733a84d2630.svg
Da-Wei Zhou
user
zhoudw
665fc9ffb99c631f4f55d34d
Yang Li
claimed_verified
2024-06-06T09:27:32.961Z
6076587d310e510df1db14bc
/avatars/afa6e30b42a6205d2a9de6b51ce4efaa.svg
liyang
user
liyang
665fc9ffb99c631f4f55d34e
Shiyin Lu
claimed_verified
2024-06-07T09:20:46.785Z
658a8a837959448ef5500ce5
https://cdn-avatars.huggingface.co/v1/production/uploads/658a8a837959448ef5500ce5/4rb2RXrsgCDH80VnwZsEt.jpeg
Shiyin Lu
user
runninglsy
665fc9ffb99c631f4f55d34f
Chao Yi
665fc9ffb99c631f4f55d350
Qing-Guo Chen
665fc9ffb99c631f4f55d351
Zhao Xu
665fc9ffb99c631f4f55d352
Weihua Luo
665fc9ffb99c631f4f55d353
Kaifu Zhang
admin_assigned
2024-06-06T08:27:52.743Z
63f87ebadf053017d1acbfdd
/avatars/e497ba5f41a2587837b4a6118d9367bb.svg
Kaifu Zhang
user
zhangkaifu314
665fc9ffb99c631f4f55d354
De-Chuan Zhan
665fc9ffb99c631f4f55d355
Han-Jia Ye
Parrot: Multilingual Visual Instruction Tuning
The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V
has marked a significant step towards artificial general intelligence. Existing
methods mainly focus on aligning vision encoders with LLMs through supervised
fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs'
inherent ability to react to multiple languages progressively deteriorate as
the training process evolves. We empirically find that the imbalanced SFT
datasets, primarily composed of English-centric image-text pairs, lead to
significantly reduced performance in non-English languages. This is due to the
failure of aligning the vision encoder and LLM with multilingual tokens during
the SFT process. In this paper, we introduce Parrot, a novel method that
utilizes textual guidance to drive visual token alignment at the language
level. Parrot makes the visual tokens condition on diverse language inputs and
uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens.
Specifically, to enhance non-English visual tokens alignment, we compute the
cross-attention using the initial visual features and textual embeddings, the
result of which is then fed into the MoE router to select the most relevant
experts. The selected experts subsequently convert the initial visual tokens
into language-specific visual tokens. Moreover, considering the current lack of
benchmarks for evaluating multilingual capabilities within the field, we
collect and make available a Massive Multilingual Multimodal Benchmark which
includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our
method not only demonstrates state-of-the-art performance on multilingual
MMBench and MMMB, but also excels across a broad range of multimodal tasks.
Both the source code and the training dataset of Parrot will be made publicly
available.
Parrot enhances multimodal language models with multilingual visual token alignment using textual guidance and Mixture-of-Experts, achieving state-of-the-art performance on multilingual benchmarks.
Multimodal Large Language Models MLLMs supervised fine-tuning SFT vision encoders multilingual tokens Parrot textual guidance visual token alignment Mixture-of-Experts MoE cross-attention language-specific visual tokens MMMB