CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large
  Language Models in 167 Languages
The driving factors behind the development of large language models (LLMs)
with impressive learning capabilities are their colossal model sizes and
extensive training datasets. Along with the progress in natural language
processing, LLMs have been frequently made accessible to the public to foster
deeper investigation and applications. However, when it comes to training
datasets for these LLMs, especially the recent state-of-the-art models, they
are often not fully disclosed. Creating training data for high-performing LLMs
involves extensive cleaning and deduplication to ensure the necessary level of
quality. The lack of transparency for training data has thus hampered research
on attributing and addressing hallucination and bias issues in LLMs, hindering
replication efforts and further advancements in the community. These challenges
become even more pronounced in multilingual learning scenarios, where the
available multilingual text datasets are often inadequately collected and
cleaned. Consequently, there is a lack of open-source and readily usable
dataset to effectively train LLMs in multiple languages. To overcome this
issue, we present CulturaX, a substantial multilingual dataset with 6.3
trillion tokens in 167 languages, tailored for LLM development. Our dataset
undergoes meticulous cleaning and deduplication through a rigorous pipeline of
multiple stages to accomplish the best quality for model training, including
language identification, URL-based filtering, metric-based cleaning, document
refinement, and data deduplication. CulturaX is fully released to the public in
HuggingFace to facilitate research and advancements in multilingual LLMs:
https://huggingface.co/datasets/uonlp/CulturaX.
65090f6e7ee07e274b2653f1
Thuat Nguyen
claimed_verified
2023-09-19T11:29:50.216Z
62b2151c6a5435fd9a68b18f
https://cdn-avatars.huggingface.co/v1/production/uploads/62b2151c6a5435fd9a68b18f/dK8eT7fcG0ZnXgk0GjcUS.jpeg
Thuat Huu Nguyen
user
nguyenhuuthuat09
65090f6e7ee07e274b2653f2
Chien Van Nguyen
claimed_verified
2023-09-21T07:25:12.397Z
63edf17df5aef7e657262605
https://cdn-avatars.huggingface.co/v1/production/uploads/63edf17df5aef7e657262605/X3WzygIyND_0T_SWRDDU4.jpeg
Nguyen Van Chien
user
chiennv
65090f6e7ee07e274b2653f3
Viet Dac Lai
claimed_verified
2023-09-20T09:51:01.229Z
63a0a29245edac9f75fc5d54
/avatars/67ba006edfd1ceddc54ea920c09c972b.svg
Viet Lai
user
laiviet
65090f6e7ee07e274b2653f4
Hieu Man
admin_assigned
2023-09-19T15:59:46.613Z
6225ce77b213da14d165fa50
/avatars/64d2307859d73343280a895ca81e3136.svg
Hieu Man
user
Hieuman
65090f6e7ee07e274b2653f5
Nghia Trung Ngo
65090f6e7ee07e274b2653f6
Franck Dernoncourt
admin_assigned
2023-09-19T16:00:04.490Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
user
Franck-Dernoncourt
65090f6e7ee07e274b2653f7
Ryan A. Rossi
admin_assigned
2023-09-19T16:00:28.474Z
62a3ab83e4dd6252344d27cd
/avatars/7ca8510f70a58dc207b104240e30c35c.svg
Ryan A. Rossi
user
ryanrossi
65090f6e7ee07e274b2653f8
Thien Huu Nguyen
claimed_verified
2023-09-25T10:52:11.098Z
64804fad8c6a3b8f11f73912
/avatars/61e37a91d4bba35fda9bf52aadd87745.svg
Thien Huu Nguyen
user
anoperson
CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large
  Language Models in 167 Languages
The driving factors behind the development of large language models (LLMs)
with impressive learning capabilities are their colossal model sizes and
extensive training datasets. Along with the progress in natural language
processing, LLMs have been frequently made accessible to the public to foster
deeper investigation and applications. However, when it comes to training
datasets for these LLMs, especially the recent state-of-the-art models, they
are often not fully disclosed. Creating training data for high-performing LLMs
involves extensive cleaning and deduplication to ensure the necessary level of
quality. The lack of transparency for training data has thus hampered research
on attributing and addressing hallucination and bias issues in LLMs, hindering
replication efforts and further advancements in the community. These challenges
become even more pronounced in multilingual learning scenarios, where the
available multilingual text datasets are often inadequately collected and
cleaned. Consequently, there is a lack of open-source and readily usable
dataset to effectively train LLMs in multiple languages. To overcome this
issue, we present CulturaX, a substantial multilingual dataset with 6.3
trillion tokens in 167 languages, tailored for LLM development. Our dataset
undergoes meticulous cleaning and deduplication through a rigorous pipeline of
multiple stages to accomplish the best quality for model training, including
language identification, URL-based filtering, metric-based cleaning, document
refinement, and data deduplication. CulturaX is fully released to the public in
HuggingFace to facilitate research and advancements in multilingual LLMs:
https://huggingface.co/datasets/uonlp/CulturaX.
CulturaX is a large multilingual dataset designed to improve the training and quality of large language models through rigorous data cleaning and deduplication.
large language models multilingual learning hallucination bias dataset cleaning deduplication language identification URL-based filtering metric-based cleaning document refinement HuggingFace