M^3IT: A Large-Scale Dataset towards Multi-Modal Multilingual
  Instruction Tuning
Instruction tuning has significantly advanced large language models (LLMs)
such as ChatGPT, enabling them to align with human instructions across diverse
tasks. However, progress in open vision-language models (VLMs) has been limited
due to the scarcity of high-quality instruction datasets. To tackle this
challenge and promote research in the vision-language field, we introduce the
Multi-Modal, Multilingual Instruction Tuning (M^3IT) dataset, designed to
optimize VLM alignment with human instructions. Our M^3IT dataset comprises
40 carefully curated datasets, including 2.4 million instances and 400 manually
written task instructions, reformatted into a vision-to-text structure. Key
tasks are translated into 80 languages with an advanced translation system,
ensuring broader accessibility. M^3IT surpasses previous datasets regarding
task coverage, instruction number and instance scale. Moreover, we develop
Ying-VLM, a VLM model trained on our M^3IT dataset, showcasing its potential
to answer complex questions requiring world knowledge, generalize to unseen
video tasks, and comprehend unseen instructions in Chinese. To encourage
further research, we have open-sourced both the dataset and trained models.
648124fdcb64a2b85a448a4c
Lei Li
claimed_verified
2023-06-09T15:35:22.824Z
6038d6d0612f5eef3cc05ea9
https://cdn-avatars.huggingface.co/v1/production/uploads/6038d6d0612f5eef3cc05ea9/ryhvAX5djQpD5OrIlZQ1f.jpeg
Lei Li
user
tobiaslee
648124fdcb64a2b85a448a4d
Yuwei Yin
claimed_verified
2023-06-09T15:34:47.642Z
64510a21f800611f94f0d9f8
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/lOeHK9Bvt3IXcB7Urx6jZ.jpeg
Yuwei Yin
user
yuweiyin
648124fdcb64a2b85a448a4e
Shicheng Li
claimed_verified
2025-06-05T12:43:00.322Z
64539588fc2b5f69e8faac76
/avatars/111441eeb0dd4d8ad2f0d3f28277952a.svg
Li Shicheng
user
lscpku
648124fdcb64a2b85a448a4f
Liang Chen
claimed_verified
2024-03-12T12:06:45.728Z
61b0a4ce1b3d95b3d1ed9251
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Wwjr26vdudX5KYVTb8Q0a.png
Liang Chen
user
leonardPKU
648124fdcb64a2b85a448a50
Peiyi Wang
claimed_verified
2023-06-08T16:07:51.271Z
6405b1ecbe1a9448a79a1707
/avatars/94bd1e9ee91be5eff14fa7f3351363e8.svg
wangpeiyi
user
peiyi9979
648124fdcb64a2b85a448a51
Shuhuai Ren
claimed_verified
2023-06-09T15:32:59.739Z
60d2e681b8448e1785bbda06
https://cdn-avatars.huggingface.co/v1/production/uploads/1624434302056-noauth.jpeg
Shuhuai Ren
user
ShuhuaiRen
648124fdcb64a2b85a448a52
Mukai Li
claimed_verified
2023-06-13T17:38:57.385Z
624561c1939c9acec2103534
/avatars/3efdf797c53d153cea7415213fb5afc7.svg
Mukai Li
user
kiaia
648124fdcb64a2b85a448a53
Yazheng Yang
648124fdcb64a2b85a448a54
Jingjing Xu
648124fdcb64a2b85a448a55
Xu Sun
648124fdcb64a2b85a448a56
Lingpeng Kong
648124fdcb64a2b85a448a57
Qi Liu
M^3IT: A Large-Scale Dataset towards Multi-Modal Multilingual
  Instruction Tuning
Instruction tuning has significantly advanced large language models (LLMs)
such as ChatGPT, enabling them to align with human instructions across diverse
tasks. However, progress in open vision-language models (VLMs) has been limited
due to the scarcity of high-quality instruction datasets. To tackle this
challenge and promote research in the vision-language field, we introduce the
Multi-Modal, Multilingual Instruction Tuning (M^3IT) dataset, designed to
optimize VLM alignment with human instructions. Our M^3IT dataset comprises
40 carefully curated datasets, including 2.4 million instances and 400 manually
written task instructions, reformatted into a vision-to-text structure. Key
tasks are translated into 80 languages with an advanced translation system,
ensuring broader accessibility. M^3IT surpasses previous datasets regarding
task coverage, instruction number and instance scale. Moreover, we develop
Ying-VLM, a VLM model trained on our M^3IT dataset, showcasing its potential
to answer complex questions requiring world knowledge, generalize to unseen
video tasks, and comprehend unseen instructions in Chinese. To encourage
further research, we have open-sourced both the dataset and trained models.
The M$^3$IT dataset and Ying-VLM model enhance vision-language models' alignment with human instructions across various languages and modalities.
M$^3$IT dataset vision-language models instruction tuning vision-to-text task coverage instruction number instance scale Ying-VLM world knowledge unseen video tasks