OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for
  LLM Training
Large language models (LLMs) have demonstrated remarkable capabilities, but
their success heavily relies on the quality of pretraining corpora. For Chinese
LLMs, the scarcity of high-quality Chinese datasets presents a significant
challenge, often limiting their performance. To address this issue, we propose
the OpenCSG Chinese Corpus, a series of high-quality datasets specifically
designed for LLM pretraining, post-training, and fine-tuning. This corpus
includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and
Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets
focus on filtered, high-quality content derived from diverse Chinese web
sources; Cosmopedia-chinese provides synthetic, textbook-style data for
knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and
diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its
high-quality text, diverse coverage across domains, and scalable, reproducible
data curation processes. Additionally, we conducted extensive experimental
analyses, including evaluations on smaller parameter models, which demonstrated
significant performance improvements in tasks such as C-Eval, showcasing the
effectiveness of the corpus for training Chinese LLMs.
67873d8945c53fa98281bb59
Yijiong Yu
claimed_verified
2025-01-15T08:47:14.751Z
6374c494958cd71fa7ea0a9d
https://cdn-avatars.huggingface.co/v1/production/uploads/6374c494958cd71fa7ea0a9d/2YCKv6tXCZXtsIOFIIXjs.png
yuyijiong
user
yuyijiong
67873d8945c53fa98281bb5a
Ziyun Dai
admin_assigned
2025-01-15T17:00:20.722Z
667ec2ffa016ced375edfae9
/avatars/73f1298569fb0667194d1ac17fb508d6.svg
Ziyun dai
user
Oliviadzy
67873d8945c53fa98281bb5b
Zekun Wang
admin_assigned
2025-01-15T17:00:34.066Z
656832dfbd65fd41ee7aa8cd
https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
Zekun Wang
user
kugwzk
67873d8945c53fa98281bb5c
Wei Wang
67873d8945c53fa98281bb5d
Ran Chen
67873d8945c53fa98281bb5e
Ji Pei
OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for
  LLM Training
Large language models (LLMs) have demonstrated remarkable capabilities, but
their success heavily relies on the quality of pretraining corpora. For Chinese
LLMs, the scarcity of high-quality Chinese datasets presents a significant
challenge, often limiting their performance. To address this issue, we propose
the OpenCSG Chinese Corpus, a series of high-quality datasets specifically
designed for LLM pretraining, post-training, and fine-tuning. This corpus
includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and
Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets
focus on filtered, high-quality content derived from diverse Chinese web
sources; Cosmopedia-chinese provides synthetic, textbook-style data for
knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and
diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its
high-quality text, diverse coverage across domains, and scalable, reproducible
data curation processes. Additionally, we conducted extensive experimental
analyses, including evaluations on smaller parameter models, which demonstrated
significant performance improvements in tasks such as C-Eval, showcasing the
effectiveness of the corpus for training Chinese LLMs.
The OpenCSG Chinese Corpus, a series of high-quality datasets, enhances the performance of Chinese LLMs through diverse and effective pretraining, post-training, and fine-tuning.
LLMs pretraining corpora high-quality datasets Fineweb-edu-chinese Fineweb-edu-chinese-v2 Cosmopedia-chinese Smoltalk-chinese knowledge-intensive training chat-format data C-Eval