title: mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
summary: Multimodal Large Language Models (mLLMs) are trained on a large amount of
  text-image data. While most mLLMs are trained on caption-like data only,
  Alayrac et al. [2022] showed that additionally training them on interleaved
  sequences of text and images can lead to the emergence of in-context learning
  capabilities. However, the dataset they used, M3W, is not public and is only in
  English. There have been attempts to reproduce their results but the released
  datasets are English-only. In contrast, current multilingual and multimodal
  datasets are either composed of caption-like only or medium-scale or fully
  private data. This limits mLLM research for the 7,000 other languages spoken in
  the world. We therefore introduce mOSCAR, to the best of our knowledge the
  first large-scale multilingual and multimodal document corpus crawled from the
  web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
  carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
  is sufficiently safe, diverse and of good quality. We additionally train two
  types of multilingual model to prove the benefits of mOSCAR: (1) a model
  trained on a subset of mOSCAR and captioning data and (2) a model train on
  captioning data only. The model additionally trained on mOSCAR shows a strong
  boost in few-shot learning performance across various multilingual image-text
  tasks and benchmarks, confirming previous findings for English-only mLLMs.
paper_authors._id: 666c0144623133f1ce4c4dd1
paper_authors.name: Matthieu Futeral
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-14T08:49:17.543Z
paper_authors._id: 638878e0c9a44f05de452e91
paper_authors.avatarUrl: /avatars/5748195a0ac761e2776548aabc3a53e3.svg
paper_authors.fullname: Matthieu Futeral
paper_authors.type: user
paper_authors.user: matthieufp
paper_authors._id: 666c0144623133f1ce4c4dd2
paper_authors.name: Armel Zebaze
paper_authors._id: 666c0144623133f1ce4c4dd3
paper_authors.name: Pedro Ortiz Suarez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:31:24.893Z
paper_authors._id: 602705869cad998710e2e8ab
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632322117476-602705869cad998710e2e8ab.jpeg
paper_authors.fullname: Pedro Ortiz Suarez
paper_authors.type: user
paper_authors.user: pjox
paper_authors._id: 666c0144623133f1ce4c4dd4
paper_authors.name: Julien Abadji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:31:32.596Z
paper_authors._id: 614b51fbf2a3f96861e548ae
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632326156496-614b51fbf2a3f96861e548ae.jpeg
paper_authors.fullname: Julien Abadji
paper_authors.type: user
paper_authors.user: uj
paper_authors._id: 666c0144623133f1ce4c4dd5
paper_authors.name: Rémi Lacroix
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:31:38.914Z
paper_authors._id: 6303483bef6f432d84e9a662
paper_authors.avatarUrl: /avatars/67aea8a53278c99126932c9c4fbee246.svg
paper_authors.fullname: Rémi Lacroix
paper_authors.type: user
paper_authors.user: rlacroix
paper_authors._id: 666c0144623133f1ce4c4dd6
paper_authors.name: Cordelia Schmid
paper_authors._id: 666c0144623133f1ce4c4dd7
paper_authors.name: Rachel Bawden
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:31:47.884Z
paper_authors._id: 613a0c85e63aa832a98cecc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1653912785529-613a0c85e63aa832a98cecc4.png
paper_authors.fullname: Rachel Bawden
paper_authors.type: user
paper_authors.user: rbawden
paper_authors._id: 666c0144623133f1ce4c4dd8
paper_authors.name: Benoît Sagot
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:31:55.196Z
paper_authors._id: 602ba2a739515f8d31237967
paper_authors.avatarUrl: /avatars/ef534712ca682bf74ec7eef17d3d1b5f.svg
paper_authors.fullname: Benoît Sagot
paper_authors.type: user
paper_authors.user: sagot
paper_title: mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus
paper_summary: Multimodal Large Language Models (mLLMs) are trained on a large amount of
  text-image data. While most mLLMs are trained on caption-like data only,
  Alayrac et al. [2022] showed that additionally training them on interleaved
  sequences of text and images can lead to the emergence of in-context learning
  capabilities. However, the dataset they used, M3W, is not public and is only in
  English. There have been attempts to reproduce their results but the released
  datasets are English-only. In contrast, current multilingual and multimodal
  datasets are either composed of caption-like only or medium-scale or fully
  private data. This limits mLLM research for the 7,000 other languages spoken in
  the world. We therefore introduce mOSCAR, to the best of our knowledge the
  first large-scale multilingual and multimodal document corpus crawled from the
  web. It covers 163 languages, 315M documents, 214B tokens and 1.2B images. We
  carefully conduct a set of filtering and evaluation steps to make sure mOSCAR
  is sufficiently safe, diverse and of good quality. We additionally train two
  types of multilingual model to prove the benefits of mOSCAR: (1) a model
  trained on a subset of mOSCAR and captioning data and (2) a model train on
  captioning data only. The model additionally trained on mOSCAR shows a strong
  boost in few-shot learning performance across various multilingual image-text
  tasks and benchmarks, confirming previous findings for English-only mLLMs.
paper_ai_summary: mOSCAR, a large-scale multilingual and multimodal document corpus, enhances few-shot learning performance in multilingual image-text tasks when used to train models.
paper_ai_keywords: Multimodal Large Language Models mLLMs caption-like data in-context learning M3W mOSCAR multilingual multimodal document corpus filtering evaluation few-shot learning multilingual image-text tasks benchmarks