title: CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for
    pre-training large language models
summary: We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a
  high-quality 500GB subset of the Chinese Corpora Internet 3.0
  (CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a
  novel two-stage hybrid filtering pipeline that significantly enhances data
  quality. To evaluate its effectiveness, we trained a 0.5B parameter model from
  scratch on 100B tokens across various datasets, achieving superior performance
  on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and
  WanjuanV1. The high-quality filtering process effectively distills the
  capabilities of the Qwen2-72B-instruct model into a compact 0.5B model,
  attaining optimal F1 scores for Chinese web data classification. We believe
  this open-access dataset will facilitate broader access to high-quality
  language models.
paper_authors._id: 671af8b682e53a391ca9398e
paper_authors.name: Liangdong Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-25T09:30:14.228Z
paper_authors._id: 63a11ce02fabbbb899a01d58
paper_authors.avatarUrl: /avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
paper_authors.fullname: ldwang
paper_authors.type: user
paper_authors.user: ldwang
paper_authors._id: 671af8b682e53a391ca9398f
paper_authors.name: Bo-Wen Zhang
paper_authors._id: 671af8b682e53a391ca93990
paper_authors.name: Chengwei Wu
paper_authors._id: 671af8b682e53a391ca93991
paper_authors.name: Hanyu Zhao
paper_authors._id: 671af8b682e53a391ca93992
paper_authors.name: Xiaofeng Shi
paper_authors._id: 671af8b682e53a391ca93993
paper_authors.name: Shuhao Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:52.017Z
paper_authors._id: 642e72cec1b0f8e4e76af16d
paper_authors.avatarUrl: /avatars/f900811d3c22a114c67283b646949f86.svg
paper_authors.fullname: shuhao gu
paper_authors.type: user
paper_authors.user: gsh33
paper_authors._id: 671af8b682e53a391ca93994
paper_authors.name: Jijie Li
paper_authors._id: 671af8b682e53a391ca93995
paper_authors.name: Quanyue Ma
paper_authors._id: 671af8b682e53a391ca93996
paper_authors.name: TengFei Pan
paper_authors._id: 671af8b682e53a391ca93997
paper_authors.name: Guang Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:24:31.441Z
paper_authors._id: 632c234f42c386ebd2710434
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
paper_authors.fullname: Guang Liu
paper_authors.type: user
paper_authors.user: ZacLiu
paper_title: CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for
    pre-training large language models
paper_summary: We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a
  high-quality 500GB subset of the Chinese Corpora Internet 3.0
  (CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a
  novel two-stage hybrid filtering pipeline that significantly enhances data
  quality. To evaluate its effectiveness, we trained a 0.5B parameter model from
  scratch on 100B tokens across various datasets, achieving superior performance
  on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and
  WanjuanV1. The high-quality filtering process effectively distills the
  capabilities of the Qwen2-72B-instruct model into a compact 0.5B model,
  attaining optimal F1 scores for Chinese web data classification. We believe
  this open-access dataset will facilitate broader access to high-quality
  language models.