title: Parrot: Multilingual Visual Instruction Tuning
summary: The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V
  has marked a significant step towards artificial general intelligence. Existing
  methods mainly focus on aligning vision encoders with LLMs through supervised
  fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs'
  inherent ability to react to multiple languages progressively deteriorate as
  the training process evolves. We empirically find that the imbalanced SFT
  datasets, primarily composed of English-centric image-text pairs, lead to
  significantly reduced performance in non-English languages. This is due to the
  failure of aligning the vision encoder and LLM with multilingual tokens during
  the SFT process. In this paper, we introduce Parrot, a novel method that
  utilizes textual guidance to drive visual token alignment at the language
  level. Parrot makes the visual tokens condition on diverse language inputs and
  uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens.
  Specifically, to enhance non-English visual tokens alignment, we compute the
  cross-attention using the initial visual features and textual embeddings, the
  result of which is then fed into the MoE router to select the most relevant
  experts. The selected experts subsequently convert the initial visual tokens
  into language-specific visual tokens. Moreover, considering the current lack of
  benchmarks for evaluating multilingual capabilities within the field, we
  collect and make available a Massive Multilingual Multimodal Benchmark which
  includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our
  method not only demonstrates state-of-the-art performance on multilingual
  MMBench and MMMB, but also excels across a broad range of multimodal tasks.
  Both the source code and the training dataset of Parrot will be made publicly
  available.
paper_authors._id: 665fc9ffb99c631f4f55d34b
paper_authors.name: Hai-Long Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-05T07:30:37.957Z
paper_authors._id: 6623975c728f756224d4b768
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6623975c728f756224d4b768/US9n0k45Y1TF6WqOhgBhZ.jpeg
paper_authors.fullname: Allen Sun
paper_authors.type: user
paper_authors.user: Allen8
paper_authors._id: 665fc9ffb99c631f4f55d34c
paper_authors.name: Da-Wei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:26:57.211Z
paper_authors._id: 64254ff8120a3ed323314e0c
paper_authors.avatarUrl: /avatars/134325256b9073a96c6d7733a84d2630.svg
paper_authors.fullname: Da-Wei Zhou
paper_authors.type: user
paper_authors.user: zhoudw
paper_authors._id: 665fc9ffb99c631f4f55d34d
paper_authors.name: Yang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-06T09:27:32.961Z
paper_authors._id: 6076587d310e510df1db14bc
paper_authors.avatarUrl: /avatars/afa6e30b42a6205d2a9de6b51ce4efaa.svg
paper_authors.fullname: liyang
paper_authors.type: user
paper_authors.user: liyang
paper_authors._id: 665fc9ffb99c631f4f55d34e
paper_authors.name: Shiyin Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-07T09:20:46.785Z
paper_authors._id: 658a8a837959448ef5500ce5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/658a8a837959448ef5500ce5/4rb2RXrsgCDH80VnwZsEt.jpeg
paper_authors.fullname: Shiyin Lu
paper_authors.type: user
paper_authors.user: runninglsy
paper_authors._id: 665fc9ffb99c631f4f55d34f
paper_authors.name: Chao Yi
paper_authors._id: 665fc9ffb99c631f4f55d350
paper_authors.name: Qing-Guo Chen
paper_authors._id: 665fc9ffb99c631f4f55d351
paper_authors.name: Zhao Xu
paper_authors._id: 665fc9ffb99c631f4f55d352
paper_authors.name: Weihua Luo
paper_authors._id: 665fc9ffb99c631f4f55d353
paper_authors.name: Kaifu Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:27:52.743Z
paper_authors._id: 63f87ebadf053017d1acbfdd
paper_authors.avatarUrl: /avatars/e497ba5f41a2587837b4a6118d9367bb.svg
paper_authors.fullname: Kaifu Zhang
paper_authors.type: user
paper_authors.user: zhangkaifu314
paper_authors._id: 665fc9ffb99c631f4f55d354
paper_authors.name: De-Chuan Zhan
paper_authors._id: 665fc9ffb99c631f4f55d355
paper_authors.name: Han-Jia Ye
paper_title: Parrot: Multilingual Visual Instruction Tuning
paper_summary: The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V
  has marked a significant step towards artificial general intelligence. Existing
  methods mainly focus on aligning vision encoders with LLMs through supervised
  fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs'
  inherent ability to react to multiple languages progressively deteriorate as
  the training process evolves. We empirically find that the imbalanced SFT
  datasets, primarily composed of English-centric image-text pairs, lead to
  significantly reduced performance in non-English languages. This is due to the
  failure of aligning the vision encoder and LLM with multilingual tokens during
  the SFT process. In this paper, we introduce Parrot, a novel method that
  utilizes textual guidance to drive visual token alignment at the language
  level. Parrot makes the visual tokens condition on diverse language inputs and
  uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens.
  Specifically, to enhance non-English visual tokens alignment, we compute the
  cross-attention using the initial visual features and textual embeddings, the
  result of which is then fed into the MoE router to select the most relevant
  experts. The selected experts subsequently convert the initial visual tokens
  into language-specific visual tokens. Moreover, considering the current lack of
  benchmarks for evaluating multilingual capabilities within the field, we
  collect and make available a Massive Multilingual Multimodal Benchmark which
  includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our
  method not only demonstrates state-of-the-art performance on multilingual
  MMBench and MMMB, but also excels across a broad range of multimodal tasks.
  Both the source code and the training dataset of Parrot will be made publicly
  available.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) supervised fine-tuning (SFT) vision encoders LLMs (Large Language Models) imbalanced SFT datasets multilingual tokens textual guidance visual token alignment language level Mixture-of-Experts (MoE) cross-attention visual features textual embeddings MoE router experts language-specific visual tokens Massive Multilingual Multimodal Benchmark (MMMMB) multilingual MMBench MMMB