title: M^3IT: A Large-Scale Dataset towards Multi-Modal Multilingual
    Instruction Tuning
summary: Instruction tuning has significantly advanced large language models (LLMs)
  such as ChatGPT, enabling them to align with human instructions across diverse
  tasks. However, progress in open vision-language models (VLMs) has been limited
  due to the scarcity of high-quality instruction datasets. To tackle this
  challenge and promote research in the vision-language field, we introduce the
  Multi-Modal, Multilingual Instruction Tuning (M^3IT) dataset, designed to
  optimize VLM alignment with human instructions. Our M^3IT dataset comprises
  40 carefully curated datasets, including 2.4 million instances and 400 manually
  written task instructions, reformatted into a vision-to-text structure. Key
  tasks are translated into 80 languages with an advanced translation system,
  ensuring broader accessibility. M^3IT surpasses previous datasets regarding
  task coverage, instruction number and instance scale. Moreover, we develop
  Ying-VLM, a VLM model trained on our M^3IT dataset, showcasing its potential
  to answer complex questions requiring world knowledge, generalize to unseen
  video tasks, and comprehend unseen instructions in Chinese. To encourage
  further research, we have open-sourced both the dataset and trained models.
paper_authors._id: 648124fdcb64a2b85a448a4c
paper_authors.name: Lei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:35:22.824Z
paper_authors._id: 6038d6d0612f5eef3cc05ea9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
paper_authors.fullname: Lei Li
paper_authors.type: user
paper_authors.user: tobiaslee
paper_authors._id: 648124fdcb64a2b85a448a4d
paper_authors.name: Yuwei Yin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:34:47.642Z
paper_authors._id: 64510a21f800611f94f0d9f8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lOeHK9Bvt3IXcB7Urx6jZ.jpeg
paper_authors.fullname: Yuwei Yin
paper_authors.type: user
paper_authors.user: yuweiyin
paper_authors._id: 648124fdcb64a2b85a448a4e
paper_authors.name: Shicheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T12:43:00.322Z
paper_authors._id: 64539588fc2b5f69e8faac76
paper_authors.avatarUrl: /avatars/111441eeb0dd4d8ad2f0d3f28277952a.svg
paper_authors.fullname: Li Shicheng
paper_authors.type: user
paper_authors.user: lscpku
paper_authors._id: 648124fdcb64a2b85a448a4f
paper_authors.name: Liang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-12T12:06:45.728Z
paper_authors._id: 61b0a4ce1b3d95b3d1ed9251
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: leonardPKU
paper_authors._id: 648124fdcb64a2b85a448a50
paper_authors.name: Peiyi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-08T16:07:51.271Z
paper_authors._id: 6405b1ecbe1a9448a79a1707
paper_authors.avatarUrl: /avatars/94bd1e9ee91be5eff14fa7f3351363e8.svg
paper_authors.fullname: wangpeiyi
paper_authors.type: user
paper_authors.user: peiyi9979
paper_authors._id: 648124fdcb64a2b85a448a51
paper_authors.name: Shuhuai Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:32:59.739Z
paper_authors._id: 60d2e681b8448e1785bbda06
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
paper_authors.fullname: Shuhuai Ren
paper_authors.type: user
paper_authors.user: ShuhuaiRen
paper_authors._id: 648124fdcb64a2b85a448a52
paper_authors.name: Mukai Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:38:57.385Z
paper_authors._id: 624561c1939c9acec2103534
paper_authors.avatarUrl: /avatars/3efdf797c53d153cea7415213fb5afc7.svg
paper_authors.fullname: Mukai Li
paper_authors.type: user
paper_authors.user: kiaia
paper_authors._id: 648124fdcb64a2b85a448a53
paper_authors.name: Yazheng Yang
paper_authors._id: 648124fdcb64a2b85a448a54
paper_authors.name: Jingjing Xu
paper_authors._id: 648124fdcb64a2b85a448a55
paper_authors.name: Xu Sun
paper_authors._id: 648124fdcb64a2b85a448a56
paper_authors.name: Lingpeng Kong
paper_authors._id: 648124fdcb64a2b85a448a57
paper_authors.name: Qi Liu
paper_title: M^3IT: A Large-Scale Dataset towards Multi-Modal Multilingual
    Instruction Tuning
paper_summary: Instruction tuning has significantly advanced large language models (LLMs)
  such as ChatGPT, enabling them to align with human instructions across diverse
  tasks. However, progress in open vision-language models (VLMs) has been limited
  due to the scarcity of high-quality instruction datasets. To tackle this
  challenge and promote research in the vision-language field, we introduce the
  Multi-Modal, Multilingual Instruction Tuning (M^3IT) dataset, designed to
  optimize VLM alignment with human instructions. Our M^3IT dataset comprises
  40 carefully curated datasets, including 2.4 million instances and 400 manually
  written task instructions, reformatted into a vision-to-text structure. Key
  tasks are translated into 80 languages with an advanced translation system,
  ensuring broader accessibility. M^3IT surpasses previous datasets regarding
  task coverage, instruction number and instance scale. Moreover, we develop
  Ying-VLM, a VLM model trained on our M^3IT dataset, showcasing its potential
  to answer complex questions requiring world knowledge, generalize to unseen
  video tasks, and comprehend unseen instructions in Chinese. To encourage
  further research, we have open-sourced both the dataset and trained models.
paper_ai_summary: The M$^3$IT dataset and Ying-VLM model enhance vision-language models' alignment with human instructions across various languages and modalities.
paper_ai_keywords: M$^3$IT dataset vision-language models instruction tuning vision-to-text task coverage instruction number instance scale Ying-VLM world knowledge unseen video tasks