title: Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
summary: In this study, we introduce CT-LLM, a 2B large language model (LLM) that
  illustrates a pivotal shift towards prioritizing the Chinese language in
  developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the
  conventional methodology by primarily incorporating Chinese textual data,
  utilizing an extensive corpus of 1,200 billion tokens, including 800 billion
  Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This
  strategic composition facilitates the model's exceptional proficiency in
  understanding and processing Chinese, a capability further enhanced through
  alignment techniques. Demonstrating remarkable performance on the CHC-Bench,
  CT-LLM excels in Chinese language tasks, and showcases its adeptness in English
  through SFT. This research challenges the prevailing paradigm of training LLMs
  predominantly on English corpora and then adapting them to other languages,
  broadening the horizons for LLM training methodologies. By open-sourcing the
  full process of training a Chinese LLM, including a detailed data processing
  procedure with the obtained Massive Appropriate Pretraining Chinese Corpus
  (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark
  (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster
  further exploration and innovation in both academia and industry, paving the
  way for more inclusive and versatile language models.
paper_authors._id: 66135b40db728ea967146837
paper_authors.name: Xinrun Du
paper_authors._id: 66135b40db728ea967146838
paper_authors.name: Zhouliang Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:20:03.663Z
paper_authors._id: 62a80fe3ac97233f1625235a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62a80fe3ac97233f1625235a/_rGtpqdY7OEBz3pyqb6fE.jpeg
paper_authors.fullname: Zhouliang Yu
paper_authors.type: user
paper_authors.user: zhouliang
paper_authors._id: 66135b40db728ea967146839
paper_authors.name: Songyang Gao
paper_authors._id: 66135b40db728ea96714683a
paper_authors.name: Ding Pan
paper_authors._id: 66135b40db728ea96714683b
paper_authors.name: Yuyang Cheng
paper_authors._id: 66135b40db728ea96714683c
paper_authors.name: Ziyang Ma
paper_authors._id: 66135b40db728ea96714683d
paper_authors.name: Ruibin Yuan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:57:35.037Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 66135b40db728ea96714683e
paper_authors.name: Xingwei Qu
paper_authors._id: 66135b40db728ea96714683f
paper_authors.name: Jiaheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:03:56.063Z
paper_authors._id: 65377c30e48353201e6fdda0
paper_authors.avatarUrl: /avatars/a8f803b6f2e598eaee9c52c0d2ddfc16.svg
paper_authors.fullname: Jiaheng Liu
paper_authors.type: user
paper_authors.user: CheeryLJH
paper_authors._id: 66135b40db728ea967146840
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:54:26.941Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 66135b40db728ea967146841
paper_authors.name: Xinchen Luo
paper_authors._id: 66135b40db728ea967146842
paper_authors.name: Guorui Zhou
paper_authors._id: 66135b40db728ea967146843
paper_authors.name: Binhang Yuan
paper_authors._id: 66135b40db728ea967146844
paper_authors.name: Wenhu Chen
paper_authors._id: 66135b40db728ea967146845
paper_authors.name: Jie Fu
paper_authors._id: 66135b40db728ea967146846
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-31T06:53:38.464Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_title: Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Model
paper_summary: In this study, we introduce CT-LLM, a 2B large language model (LLM) that
  illustrates a pivotal shift towards prioritizing the Chinese language in
  developing LLMs. Uniquely initiated from scratch, CT-LLM diverges from the
  conventional methodology by primarily incorporating Chinese textual data,
  utilizing an extensive corpus of 1,200 billion tokens, including 800 billion
  Chinese tokens, 300 billion English tokens, and 100 billion code tokens. This
  strategic composition facilitates the model's exceptional proficiency in
  understanding and processing Chinese, a capability further enhanced through
  alignment techniques. Demonstrating remarkable performance on the CHC-Bench,
  CT-LLM excels in Chinese language tasks, and showcases its adeptness in English
  through SFT. This research challenges the prevailing paradigm of training LLMs
  predominantly on English corpora and then adapting them to other languages,
  broadening the horizons for LLM training methodologies. By open-sourcing the
  full process of training a Chinese LLM, including a detailed data processing
  procedure with the obtained Massive Appropriate Pretraining Chinese Corpus
  (MAP-CC), a well-chosen multidisciplinary Chinese Hard Case Benchmark
  (CHC-Bench), and the 2B-size Chinese Tiny LLM (CT-LLM), we aim to foster
  further exploration and innovation in both academia and industry, paving the
  way for more inclusive and versatile language models.
paper_ai_summary: CT-LLM, a large language model emphasizing Chinese language training, demonstrates exceptional proficiency through extensive Chinese textual data and outperforms in both Chinese and English tasks.
paper_ai_keywords: large language model (LLM) Chinese language CHC-Bench SFT MAP-CC Chinese Tiny LLM (CT-LLM)