title: OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for
    LLM Training
summary: Large language models (LLMs) have demonstrated remarkable capabilities, but
  their success heavily relies on the quality of pretraining corpora. For Chinese
  LLMs, the scarcity of high-quality Chinese datasets presents a significant
  challenge, often limiting their performance. To address this issue, we propose
  the OpenCSG Chinese Corpus, a series of high-quality datasets specifically
  designed for LLM pretraining, post-training, and fine-tuning. This corpus
  includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and
  Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets
  focus on filtered, high-quality content derived from diverse Chinese web
  sources; Cosmopedia-chinese provides synthetic, textbook-style data for
  knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and
  diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its
  high-quality text, diverse coverage across domains, and scalable, reproducible
  data curation processes. Additionally, we conducted extensive experimental
  analyses, including evaluations on smaller parameter models, which demonstrated
  significant performance improvements in tasks such as C-Eval, showcasing the
  effectiveness of the corpus for training Chinese LLMs.
paper_authors._id: 67873d8945c53fa98281bb59
paper_authors.name: Yijiong Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:47:14.751Z
paper_authors._id: 6374c494958cd71fa7ea0a9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6374c494958cd71fa7ea0a9d/2YCKv6tXCZXtsIOFIIXjs.png
paper_authors.fullname: yuyijiong
paper_authors.type: user
paper_authors.user: yuyijiong
paper_authors._id: 67873d8945c53fa98281bb5a
paper_authors.name: Ziyun Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:00:20.722Z
paper_authors._id: 667ec2ffa016ced375edfae9
paper_authors.avatarUrl: /avatars/73f1298569fb0667194d1ac17fb508d6.svg
paper_authors.fullname: Ziyun dai
paper_authors.type: user
paper_authors.user: Oliviadzy
paper_authors._id: 67873d8945c53fa98281bb5b
paper_authors.name: Zekun Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-15T17:00:34.066Z
paper_authors._id: 656832dfbd65fd41ee7aa8cd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656832dfbd65fd41ee7aa8cd/HHkyetTqNq1wIBPipzjQA.jpeg
paper_authors.fullname: Zekun Wang
paper_authors.type: user
paper_authors.user: kugwzk
paper_authors._id: 67873d8945c53fa98281bb5c
paper_authors.name: Wei Wang
paper_authors._id: 67873d8945c53fa98281bb5d
paper_authors.name: Ran Chen
paper_authors._id: 67873d8945c53fa98281bb5e
paper_authors.name: Ji Pei
paper_title: OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for
    LLM Training
paper_summary: Large language models (LLMs) have demonstrated remarkable capabilities, but
  their success heavily relies on the quality of pretraining corpora. For Chinese
  LLMs, the scarcity of high-quality Chinese datasets presents a significant
  challenge, often limiting their performance. To address this issue, we propose
  the OpenCSG Chinese Corpus, a series of high-quality datasets specifically
  designed for LLM pretraining, post-training, and fine-tuning. This corpus
  includes Fineweb-edu-chinese, Fineweb-edu-chinese-v2, Cosmopedia-chinese, and
  Smoltalk-chinese, each with distinct characteristics: Fineweb-edu datasets
  focus on filtered, high-quality content derived from diverse Chinese web
  sources; Cosmopedia-chinese provides synthetic, textbook-style data for
  knowledge-intensive training; and Smoltalk-chinese emphasizes stylistic and
  diverse chat-format data. The OpenCSG Chinese Corpus is characterized by its
  high-quality text, diverse coverage across domains, and scalable, reproducible
  data curation processes. Additionally, we conducted extensive experimental
  analyses, including evaluations on smaller parameter models, which demonstrated
  significant performance improvements in tasks such as C-Eval, showcasing the
  effectiveness of the corpus for training Chinese LLMs.
paper_ai_keywords: Large language models (LLMs) OpenCSG Chinese Corpus Fineweb-edu-chinese Fineweb-edu-chinese-v2 Cosmopedia-chinese Smoltalk-chinese C-Eval parameter-efficient fine-tuning