title: GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for
    Minority Languages
summary: The need for large text corpora has increased with the advent of pretrained
  language models and, in particular, the discovery of scaling laws for these
  models. Most available corpora have sufficient data only for languages with
  large dominant communities. However, there is no corpus available that (i)
  covers a wide range of minority languages; (ii) is generated by an open-source
  reproducible pipeline; and (iii) is rigorously cleaned from noise, making it
  trustworthy to use. We present GlotCC, a clean, document-level, 2TB general
  domain corpus derived from CommonCrawl, covering more than 1000 languages. We
  make GlotCC and the system used to generate it - including the pipeline,
  language identification model, and filters - available to the research
  community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1,
  Pipeline v. 3.0 https://github.com/cisnlp/GlotCC.
paper_authors._id: 67248f1a708f76fcb928a405
paper_authors.name: Amir Hossein Kargaran
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-03T14:21:50.418Z
paper_authors._id: 61bf84c8ca59d6d196a1b4e8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61bf84c8ca59d6d196a1b4e8/L_NvUwlMYcye9X35z6f7e.jpeg
paper_authors.fullname: Amir Hossein Kargaran
paper_authors.type: user
paper_authors.user: kargaranamir
paper_authors._id: 67248f1a708f76fcb928a406
paper_authors.name: François Yvon
paper_authors._id: 67248f1a708f76fcb928a407
paper_authors.name: Hinrich Schütze
paper_title: GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for
    Minority Languages
paper_summary: The need for large text corpora has increased with the advent of pretrained
  language models and, in particular, the discovery of scaling laws for these
  models. Most available corpora have sufficient data only for languages with
  large dominant communities. However, there is no corpus available that (i)
  covers a wide range of minority languages; (ii) is generated by an open-source
  reproducible pipeline; and (iii) is rigorously cleaned from noise, making it
  trustworthy to use. We present GlotCC, a clean, document-level, 2TB general
  domain corpus derived from CommonCrawl, covering more than 1000 languages. We
  make GlotCC and the system used to generate it - including the pipeline,
  language identification model, and filters - available to the research
  community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1,
  Pipeline v. 3.0 https://github.com/cisnlp/GlotCC.
paper_ai_summary: GlotCC is a comprehensive, high-quality corpus covering over 1000 languages, derived from CommonCrawl, and includes reproducible pipelines for generation, language identification, and cleaning.
paper_ai_keywords: language identification model document-level corpus reproducible pipeline CommonCrawl corpus cleaning