title: YuLan-Mini: An Open Data-efficient Language Model
summary: Effective pre-training of large language models (LLMs) has been challenging
  due to the immense resource demands and the complexity of the technical
  processes involved. This paper presents a detailed technical report on
  YuLan-Mini, a highly capable base model with 2.42B parameters that achieves
  top-tier performance among models of similar parameter scale. Our pre-training
  approach focuses on enhancing training efficacy through three key technical
  contributions: an elaborate data pipeline combines data cleaning with data
  schedule strategies, a robust optimization method to mitigate training
  instability, and an effective annealing approach that incorporates targeted
  data selection and long context training. Remarkably, YuLan-Mini, trained on
  1.08T tokens, achieves performance comparable to industry-leading models that
  require significantly more data. To facilitate reproduction, we release the
  full details of the data composition for each training phase. Project details
  can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.
paper_authors._id: 676d2c63310ca4eb397415fe
paper_authors.name: Yiwen Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:31:52.455Z
paper_authors._id: 6317419f3eb2544b62389a79
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6317419f3eb2544b62389a79/8oU90du902ATtBPYCcLFK.jpeg
paper_authors.fullname: Ivan Hu
paper_authors.type: user
paper_authors.user: IvanHU
paper_authors._id: 676d2c63310ca4eb397415ff
paper_authors.name: Huatong Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:02:13.538Z
paper_authors._id: 66163dc8c7f45b3f893ff40b
paper_authors.avatarUrl: /avatars/801043dac0caae90bbca8c9d3e2e203b.svg
paper_authors.fullname: Song Huatong
paper_authors.type: user
paper_authors.user: XXsongLALA
paper_authors._id: 676d2c63310ca4eb39741600
paper_authors.name: Jia Deng
paper_authors._id: 676d2c63310ca4eb39741601
paper_authors.name: Jiapeng Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:01:54.842Z
paper_authors._id: 61482b08c3ca9182d68debe9
paper_authors.avatarUrl: /avatars/4319add48a092adfee3bc45930ac6d2a.svg
paper_authors.fullname: Jiapeng Wang
paper_authors.type: user
paper_authors.user: jpWang
paper_authors._id: 676d2c63310ca4eb39741602
paper_authors.name: Jie Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T12:46:37.345Z
paper_authors._id: 651a29d566e78720a78317ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/651a29d566e78720a78317ec/agx5hmgWBvnsEcsCPXec7.jpeg
paper_authors.fullname: Jie Chen
paper_authors.type: user
paper_authors.user: survivi
paper_authors._id: 676d2c63310ca4eb39741603
paper_authors.name: Kun Zhou
paper_authors._id: 676d2c63310ca4eb39741604
paper_authors.name: Yutao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:01:41.536Z
paper_authors._id: 625e62452a7279d3c77b5c38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/625e62452a7279d3c77b5c38/zJINew6U4_Gup4WTobb-0.jpeg
paper_authors.fullname: Yutao Zhu
paper_authors.type: user
paper_authors.user: yutaozhu94
paper_authors._id: 676d2c63310ca4eb39741605
paper_authors.name: Jinhao Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:01:35.432Z
paper_authors._id: 61b8405b516a20acdf3b85ff
paper_authors.avatarUrl: /avatars/3d2eae7c163a80b73260087b05a4230b.svg
paper_authors.fullname: Jinhao Jiang
paper_authors.type: user
paper_authors.user: Boru
paper_authors._id: 676d2c63310ca4eb39741606
paper_authors.name: Zican Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:01:29.767Z
paper_authors._id: 65180fa499cf1a4e3913a31b
paper_authors.avatarUrl: /avatars/a321b308336bb881b11b396e210226cf.svg
paper_authors.fullname: zican dong
paper_authors.type: user
paper_authors.user: cjgs20017
paper_authors._id: 676d2c63310ca4eb39741607
paper_authors.name: Wayne Xin Zhao
paper_authors._id: 676d2c63310ca4eb39741608
paper_authors.name: Ji-Rong Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-30T21:01:14.755Z
paper_authors._id: 64b8c89052b7353d8c6a1013
paper_authors.avatarUrl: /avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
paper_authors.fullname: Ji-Rong Wen
paper_authors.type: user
paper_authors.user: jrwen
paper_title: YuLan-Mini: An Open Data-efficient Language Model
paper_summary: Effective pre-training of large language models (LLMs) has been challenging
  due to the immense resource demands and the complexity of the technical
  processes involved. This paper presents a detailed technical report on
  YuLan-Mini, a highly capable base model with 2.42B parameters that achieves
  top-tier performance among models of similar parameter scale. Our pre-training
  approach focuses on enhancing training efficacy through three key technical
  contributions: an elaborate data pipeline combines data cleaning with data
  schedule strategies, a robust optimization method to mitigate training
  instability, and an effective annealing approach that incorporates targeted
  data selection and long context training. Remarkably, YuLan-Mini, trained on
  1.08T tokens, achieves performance comparable to industry-leading models that
  require significantly more data. To facilitate reproduction, we release the
  full details of the data composition for each training phase. Project details
  can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.
paper_ai_summary: YuLan-Mini, a 2.42B parameter base model, achieves top-tier performance with efficient pre-training techniques, including a data pipeline with cleaning and scheduling, robust optimization, and annealing with targeted data selection.
paper_ai_keywords: data pipeline data cleaning data schedule strategies robust optimization annealing approach targeted data selection long context training