CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for
  pre-training large language models
We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a
high-quality 500GB subset of the Chinese Corpora Internet 3.0
(CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a
novel two-stage hybrid filtering pipeline that significantly enhances data
quality. To evaluate its effectiveness, we trained a 0.5B parameter model from
scratch on 100B tokens across various datasets, achieving superior performance
on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and
WanjuanV1. The high-quality filtering process effectively distills the
capabilities of the Qwen2-72B-instruct model into a compact 0.5B model,
attaining optimal F1 scores for Chinese web data classification. We believe
this open-access dataset will facilitate broader access to high-quality
language models.
671af8b682e53a391ca9398e
Liangdong Wang
claimed_verified
2024-10-25T09:30:14.228Z
63a11ce02fabbbb899a01d58
/avatars/ee3d4088b6d32b2c18b8be91913e90dd.svg
ldwang
user
ldwang
671af8b682e53a391ca9398f
Bo-Wen Zhang
671af8b682e53a391ca93990
Chengwei Wu
671af8b682e53a391ca93991
Hanyu Zhao
671af8b682e53a391ca93992
Xiaofeng Shi
claimed_verified
2025-06-17T13:07:40.708Z
642f6c64f945a8a5c9ee5b5d
https://cdn-avatars.huggingface.co/v1/production/uploads/642f6c64f945a8a5c9ee5b5d/V_4S_39gZc3ttiO4rXccj.png
XiaofengShi
user
MonteXiaofeng
671af8b682e53a391ca93993
Shuhao Gu
claimed_verified
2024-10-30T09:54:52.017Z
642e72cec1b0f8e4e76af16d
/avatars/f900811d3c22a114c67283b646949f86.svg
shuhao gu
user
gsh33
671af8b682e53a391ca93994
Jijie Li
671af8b682e53a391ca93995
Quanyue Ma
671af8b682e53a391ca93996
TengFei Pan
671af8b682e53a391ca93997
Guang Liu
claimed_verified
2024-10-28T11:24:31.441Z
632c234f42c386ebd2710434
https://cdn-avatars.huggingface.co/v1/production/uploads/632c234f42c386ebd2710434/HyWRWi063S69JTy_IMjoe.jpeg
Guang Liu
user
ZacLiu
CCI3.0-HQ: a large-scale Chinese dataset of high quality designed for
  pre-training large language models
We present CCI3.0-HQ (https://huggingface.co/datasets/BAAI/CCI3-HQ), a
high-quality 500GB subset of the Chinese Corpora Internet 3.0
(CCI3.0)(https://huggingface.co/datasets/BAAI/CCI3-Data), developed using a
novel two-stage hybrid filtering pipeline that significantly enhances data
quality. To evaluate its effectiveness, we trained a 0.5B parameter model from
scratch on 100B tokens across various datasets, achieving superior performance
on 10 benchmarks in a zero-shot setting compared to CCI3.0, SkyPile, and
WanjuanV1. The high-quality filtering process effectively distills the
capabilities of the Qwen2-72B-instruct model into a compact 0.5B model,
attaining optimal F1 scores for Chinese web data classification. We believe
this open-access dataset will facilitate broader access to high-quality
language models.
A high-quality dataset CCI3.0-HQ is presented, which improves data quality through a novel filtering pipeline and demonstrates superior performance in zero-shot evaluations compared to other datasets.
two-stage hybrid filtering pipeline Qwen2-72B-instruct model F1 scores