title: MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
summary: We introduce MADLAD-400, a manually audited, general domain 3T token
  monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss
  the limitations revealed by self-auditing MADLAD-400, and the role data
  auditing had in the dataset creation process. We then train and release a
  10.7B-parameter multilingual machine translation model on 250 billion tokens
  covering over 450 languages using publicly available data, and find that it is
  competitive with models that are significantly larger, and report the results
  on different domains. In addition, we train a 8B-parameter language model, and
  assess the results on few-shot translation. We make the baseline models
  available to the research community.
paper_authors._id: 64ffc9041e14749e84de6190
paper_authors.name: Sneha Kudugunta
paper_authors._id: 64ffc9041e14749e84de6191
paper_authors.name: Isaac Caswell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:09:19.931Z
paper_authors._id: 617c7165e1dfc7bbb3375abe
paper_authors.avatarUrl: /avatars/f628fb21c35eaaec5ff9519c82848c44.svg
paper_authors.fullname: Isaac Caswell
paper_authors.type: user
paper_authors.user: icaswell
paper_authors._id: 64ffc9041e14749e84de6192
paper_authors.name: Biao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:25:48.067Z
paper_authors._id: 6346a600bcb5e67902ac6f44
paper_authors.avatarUrl: /avatars/cf42043c19990bb388223a61c7a1e292.svg
paper_authors.fullname: Biao Zhang
paper_authors.type: user
paper_authors.user: bzhanggo
paper_authors._id: 64ffc9041e14749e84de6193
paper_authors.name: Xavier Garcia
paper_authors._id: 64ffc9041e14749e84de6194
paper_authors.name: Christopher A. Choquette-Choo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:08:59.947Z
paper_authors._id: 649c5307e89a7650158f4965
paper_authors.avatarUrl: /avatars/3c71b501db27c86ed686ae26ad1391cd.svg
paper_authors.fullname: Christopher A. Choquette-Choo
paper_authors.type: user
paper_authors.user: cchoquette
paper_authors._id: 64ffc9041e14749e84de6195
paper_authors.name: Katherine Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:09:09.147Z
paper_authors._id: 612a1e924f3c2ac50a06f950
paper_authors.avatarUrl: /avatars/d424605e0d48327082114b878b91b161.svg
paper_authors.fullname: Katherine Lee
paper_authors.type: user
paper_authors.user: katherine1ee
paper_authors._id: 64ffc9041e14749e84de6196
paper_authors.name: Derrick Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:09:17.312Z
paper_authors._id: 634cea1605f736dff37abe84
paper_authors.avatarUrl: /avatars/6c199de4b6e7fd1320f47eeecd1310f2.svg
paper_authors.fullname: Derrick Xin
paper_authors.type: user
paper_authors.user: Derr
paper_authors._id: 64ffc9041e14749e84de6197
paper_authors.name: Aditya Kusupati
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:45:39.295Z
paper_authors._id: 65dd3b03f0f26b277be1e91e
paper_authors.avatarUrl: /avatars/67160c2d5241aea5c5d9b211f21f9caa.svg
paper_authors.fullname: Aditya Kusupati
paper_authors.type: user
paper_authors.user: adityakusupati
paper_authors._id: 64ffc9041e14749e84de6198
paper_authors.name: Romi Stella
paper_authors._id: 64ffc9041e14749e84de6199
paper_authors.name: Ankur Bapna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T12:09:39.560Z
paper_authors._id: 62cdba279403ec4719d384f8
paper_authors.avatarUrl: /avatars/c62c7d16b5cf382cdeb7fbfb5e291917.svg
paper_authors.fullname: Ankur Bapna
paper_authors.type: user
paper_authors.user: ankurbpn
paper_authors._id: 64ffc9041e14749e84de619a
paper_authors.name: Orhan Firat
paper_title: MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
paper_summary: We introduce MADLAD-400, a manually audited, general domain 3T token
  monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss
  the limitations revealed by self-auditing MADLAD-400, and the role data
  auditing had in the dataset creation process. We then train and release a
  10.7B-parameter multilingual machine translation model on 250 billion tokens
  covering over 450 languages using publicly available data, and find that it is
  competitive with models that are significantly larger, and report the results
  on different domains. In addition, we train a 8B-parameter language model, and
  assess the results on few-shot translation. We make the baseline models
  available to the research community.
paper_ai_summary: A multilingual translation model trained on a large dataset of 250 billion tokens across over 450 languages outperforms larger models, with competitive results in various domains and effective few-shot translation performance.
paper_ai_keywords: multilingual machine translation few-shot translation CommonCrawl