title: Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building
    a Chinese-Centric LLM
summary: Steel-LLM is a Chinese-centric language model developed from scratch with the
  goal of creating a high-quality, open-source model despite limited
  computational resources. Launched in March 2024, the project aimed to train a
  1-billion-parameter model on a large-scale dataset, prioritizing transparency
  and the sharing of practical insights to assist others in the community. The
  training process primarily focused on Chinese data, with a small proportion of
  English data included, addressing gaps in existing open-source LLMs by
  providing a more detailed and practical account of the model-building journey.
  Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL
  and CMMLU, outperforming early models from larger institutions. This paper
  provides a comprehensive summary of the project's key contributions, including
  data collection, model design, training methodologies, and the challenges
  encountered along the way, offering a valuable resource for researchers and
  practitioners looking to develop their own LLMs. The model checkpoints and
  training script are available at https://github.com/zhanshijinwat/Steel-LLM.
paper_authors._id: 67aac0ba91e6f5eb5476ea76
paper_authors.name: Qingshui Gu
paper_authors._id: 67aac0ba91e6f5eb5476ea77
paper_authors.name: Shu Li
paper_authors._id: 67aac0ba91e6f5eb5476ea78
paper_authors.name: Tianyu Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T07:55:15.968Z
paper_authors._id: 64ab99dcb76bfd863eba64c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64ab99dcb76bfd863eba64c1/UBXwDPx17X-gl-SzBPvrc.jpeg
paper_authors.fullname: TY.Zheng
paper_authors.type: user
paper_authors.user: aaabiao
paper_authors._id: 67aac0ba91e6f5eb5476ea79
paper_authors.name: Zhaoxiang Zhang
paper_title: Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building
    a Chinese-Centric LLM
paper_summary: Steel-LLM is a Chinese-centric language model developed from scratch with the
  goal of creating a high-quality, open-source model despite limited
  computational resources. Launched in March 2024, the project aimed to train a
  1-billion-parameter model on a large-scale dataset, prioritizing transparency
  and the sharing of practical insights to assist others in the community. The
  training process primarily focused on Chinese data, with a small proportion of
  English data included, addressing gaps in existing open-source LLMs by
  providing a more detailed and practical account of the model-building journey.
  Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL
  and CMMLU, outperforming early models from larger institutions. This paper
  provides a comprehensive summary of the project's key contributions, including
  data collection, model design, training methodologies, and the challenges
  encountered along the way, offering a valuable resource for researchers and
  practitioners looking to develop their own LLMs. The model checkpoints and
  training script are available at https://github.com/zhanshijinwat/Steel-LLM.
paper_ai_summary: A Chinese-centric 1-billion-parameter language model built from scratch with limited resources achieved competitive performance on benchmarks like CEVAL and CMMLU.
paper_ai_keywords: landmark model model-building journey data collection model design training methodologies benchmarks CEVAL CMMLU