title: CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large
    Language Models in 167 Languages
summary: The driving factors behind the development of large language models (LLMs)
  with impressive learning capabilities are their colossal model sizes and
  extensive training datasets. Along with the progress in natural language
  processing, LLMs have been frequently made accessible to the public to foster
  deeper investigation and applications. However, when it comes to training
  datasets for these LLMs, especially the recent state-of-the-art models, they
  are often not fully disclosed. Creating training data for high-performing LLMs
  involves extensive cleaning and deduplication to ensure the necessary level of
  quality. The lack of transparency for training data has thus hampered research
  on attributing and addressing hallucination and bias issues in LLMs, hindering
  replication efforts and further advancements in the community. These challenges
  become even more pronounced in multilingual learning scenarios, where the
  available multilingual text datasets are often inadequately collected and
  cleaned. Consequently, there is a lack of open-source and readily usable
  dataset to effectively train LLMs in multiple languages. To overcome this
  issue, we present CulturaX, a substantial multilingual dataset with 6.3
  trillion tokens in 167 languages, tailored for LLM development. Our dataset
  undergoes meticulous cleaning and deduplication through a rigorous pipeline of
  multiple stages to accomplish the best quality for model training, including
  language identification, URL-based filtering, metric-based cleaning, document
  refinement, and data deduplication. CulturaX is fully released to the public in
  HuggingFace to facilitate research and advancements in multilingual LLMs:
  https://huggingface.co/datasets/uonlp/CulturaX.
paper_authors._id: 65090f6e7ee07e274b2653f1
paper_authors.name: Thuat Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-19T11:29:50.216Z
paper_authors._id: 62b2151c6a5435fd9a68b18f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b2151c6a5435fd9a68b18f/dK8eT7fcG0ZnXgk0GjcUS.jpeg
paper_authors.fullname: Thuat Huu Nguyen
paper_authors.type: user
paper_authors.user: nguyenhuuthuat09
paper_authors._id: 65090f6e7ee07e274b2653f2
paper_authors.name: Chien Van Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-21T07:25:12.397Z
paper_authors._id: 63edf17df5aef7e657262605
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63edf17df5aef7e657262605/X3WzygIyND_0T_SWRDDU4.jpeg
paper_authors.fullname: Nguyen Van Chien
paper_authors.type: user
paper_authors.user: chiennv
paper_authors._id: 65090f6e7ee07e274b2653f3
paper_authors.name: Viet Dac Lai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-20T09:51:01.229Z
paper_authors._id: 63a0a29245edac9f75fc5d54
paper_authors.avatarUrl: /avatars/67ba006edfd1ceddc54ea920c09c972b.svg
paper_authors.fullname: Viet Lai
paper_authors.type: user
paper_authors.user: laiviet
paper_authors._id: 65090f6e7ee07e274b2653f4
paper_authors.name: Hieu Man
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-19T15:59:46.613Z
paper_authors._id: 6225ce77b213da14d165fa50
paper_authors.avatarUrl: /avatars/64d2307859d73343280a895ca81e3136.svg
paper_authors.fullname: Hieu Man
paper_authors.type: user
paper_authors.user: Hieuman
paper_authors._id: 65090f6e7ee07e274b2653f5
paper_authors.name: Nghia Trung Ngo
paper_authors._id: 65090f6e7ee07e274b2653f6
paper_authors.name: Franck Dernoncourt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-19T16:00:04.490Z
paper_authors._id: 62c5947524171688a9feb992
paper_authors.avatarUrl: /avatars/5a151713b9eae8dc566f5957acee3475.svg
paper_authors.fullname: Franck Dernoncourt
paper_authors.type: user
paper_authors.user: Franck-Dernoncourt
paper_authors._id: 65090f6e7ee07e274b2653f7
paper_authors.name: Ryan A. Rossi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-19T16:00:28.474Z
paper_authors._id: 62a3ab83e4dd6252344d27cd
paper_authors.avatarUrl: /avatars/7ca8510f70a58dc207b104240e30c35c.svg
paper_authors.fullname: Ryan A. Rossi
paper_authors.type: user
paper_authors.user: ryanrossi
paper_authors._id: 65090f6e7ee07e274b2653f8
paper_authors.name: Thien Huu Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-25T10:52:11.098Z
paper_authors._id: 64804fad8c6a3b8f11f73912
paper_authors.avatarUrl: /avatars/61e37a91d4bba35fda9bf52aadd87745.svg
paper_authors.fullname: Thien Huu Nguyen
paper_authors.type: user
paper_authors.user: anoperson
paper_title: CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large
    Language Models in 167 Languages
paper_summary: The driving factors behind the development of large language models (LLMs)
  with impressive learning capabilities are their colossal model sizes and
  extensive training datasets. Along with the progress in natural language
  processing, LLMs have been frequently made accessible to the public to foster
  deeper investigation and applications. However, when it comes to training
  datasets for these LLMs, especially the recent state-of-the-art models, they
  are often not fully disclosed. Creating training data for high-performing LLMs
  involves extensive cleaning and deduplication to ensure the necessary level of
  quality. The lack of transparency for training data has thus hampered research
  on attributing and addressing hallucination and bias issues in LLMs, hindering
  replication efforts and further advancements in the community. These challenges
  become even more pronounced in multilingual learning scenarios, where the
  available multilingual text datasets are often inadequately collected and
  cleaned. Consequently, there is a lack of open-source and readily usable
  dataset to effectively train LLMs in multiple languages. To overcome this
  issue, we present CulturaX, a substantial multilingual dataset with 6.3
  trillion tokens in 167 languages, tailored for LLM development. Our dataset
  undergoes meticulous cleaning and deduplication through a rigorous pipeline of
  multiple stages to accomplish the best quality for model training, including
  language identification, URL-based filtering, metric-based cleaning, document
  refinement, and data deduplication. CulturaX is fully released to the public in
  HuggingFace to facilitate research and advancements in multilingual LLMs:
  https://huggingface.co/datasets/uonlp/CulturaX.
paper_ai_keywords: large language models (LLMs) natural language processing training datasets hallucination bias multilingual learning multilingual text datasets CulturaX language identification URL-based filtering metric-based cleaning document refinement data deduplication HuggingFace