mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
Multimodal Large Language Models (mLLMs) are trained on a large amount of
text-image data. While most mLLMs are trained on caption-like data only,
Alayrac et al. [2022] showed that additionally training them on interleaved
sequences of text and images can lead to the emergence of in-context learning
capabilities. However, the dataset they used, M3W, is not public and is only in
English. There have been attempts to reproduce their results but the released
datasets are English-only. In contrast, current multilingual and multimodal
datasets are either composed of caption-like only or medium-scale or fully
private data. This limits mLLM research for the 7,000 other languages spoken in
the world. We therefore introduce mOSCAR, to the best of our knowledge the
first large-scale multilingual and multimodal document corpus crawled from the
web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
is sufficiently safe, diverse and of good quality. We additionally train two
types of multilingual model to prove the benefits of mOSCAR: (1) a model
trained on a subset of mOSCAR and captioning data and (2) a model train on
captioning data only. The model additionally trained on mOSCAR shows a strong
boost in few-shot learning performance across various multilingual image-text
tasks and benchmarks, confirming previous findings for English-only mLLMs.
666c0144623133f1ce4c4dd1
Matthieu Futeral
extracted_confirmed
2024-06-14T08:49:17.543Z
638878e0c9a44f05de452e91
/avatars/5748195a0ac761e2776548aabc3a53e3.svg
Matthieu Futeral
user
matthieufp
666c0144623133f1ce4c4dd2
Armel Zebaze
666c0144623133f1ce4c4dd3
Pedro Ortiz Suarez
admin_assigned
2024-06-14T11:31:24.893Z
602705869cad998710e2e8ab
https://cdn-avatars.huggingface.co/v1/production/uploads/1632322117476-602705869cad998710e2e8ab.jpeg
Pedro Ortiz Suarez
user
pjox
666c0144623133f1ce4c4dd4
Julien Abadji
admin_assigned
2024-06-14T11:31:32.596Z
614b51fbf2a3f96861e548ae
https://cdn-avatars.huggingface.co/v1/production/uploads/1632326156496-614b51fbf2a3f96861e548ae.jpeg
Julien Abadji
user
uj
666c0144623133f1ce4c4dd5
Rémi Lacroix
admin_assigned
2024-06-14T11:31:38.914Z
6303483bef6f432d84e9a662
/avatars/67aea8a53278c99126932c9c4fbee246.svg
Rémi Lacroix
user
rlacroix
666c0144623133f1ce4c4dd6
Cordelia Schmid
666c0144623133f1ce4c4dd7
Rachel Bawden
admin_assigned
2024-06-14T11:31:47.884Z
613a0c85e63aa832a98cecc4
https://cdn-avatars.huggingface.co/v1/production/uploads/1653912785529-613a0c85e63aa832a98cecc4.png
Rachel Bawden
user
rbawden
666c0144623133f1ce4c4dd8
Benoît Sagot
admin_assigned
2024-06-14T11:31:55.196Z
602ba2a739515f8d31237967
/avatars/ef534712ca682bf74ec7eef17d3d1b5f.svg
Benoît Sagot
user
sagot
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
Multimodal Large Language Models (mLLMs) are trained on a large amount of
text-image data. While most mLLMs are trained on caption-like data only,
Alayrac et al. [2022] showed that additionally training them on interleaved
sequences of text and images can lead to the emergence of in-context learning
capabilities. However, the dataset they used, M3W, is not public and is only in
English. There have been attempts to reproduce their results but the released
datasets are English-only. In contrast, current multilingual and multimodal
datasets are either composed of caption-like only or medium-scale or fully
private data. This limits mLLM research for the 7,000 other languages spoken in
the world. We therefore introduce mOSCAR, to the best of our knowledge the
first large-scale multilingual and multimodal document corpus crawled from the
web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
is sufficiently safe, diverse and of good quality. We additionally train two
types of multilingual model to prove the benefits of mOSCAR: (1) a model
trained on a subset of mOSCAR and captioning data and (2) a model train on
captioning data only. The model additionally trained on mOSCAR shows a strong
boost in few-shot learning performance across various multilingual image-text
tasks and benchmarks, confirming previous findings for English-only mLLMs.
mOSCAR, a large-scale multilingual and multimodal document corpus, enhances few-shot learning performance in multilingual image-text tasks when used to train models.
Multimodal Large Language Models mLLMs caption-like data in-context learning M3W mOSCAR multilingual multimodal document corpus filtering evaluation few-shot learning multilingual image-text tasks benchmarks