YuLan-Mini: An Open Data-efficient Language Model
Effective pre-training of large language models (LLMs) has been challenging
due to the immense resource demands and the complexity of the technical
processes involved. This paper presents a detailed technical report on
YuLan-Mini, a highly capable base model with 2.42B parameters that achieves
top-tier performance among models of similar parameter scale. Our pre-training
approach focuses on enhancing training efficacy through three key technical
contributions: an elaborate data pipeline combines data cleaning with data
schedule strategies, a robust optimization method to mitigate training
instability, and an effective annealing approach that incorporates targeted
data selection and long context training. Remarkably, YuLan-Mini, trained on
1.08T tokens, achieves performance comparable to industry-leading models that
require significantly more data. To facilitate reproduction, we release the
full details of the data composition for each training phase. Project details
can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.
676d2c63310ca4eb397415fe
Yiwen Hu
claimed_verified
2024-12-30T19:31:52.455Z
6317419f3eb2544b62389a79
https://cdn-avatars.huggingface.co/v1/production/uploads/6317419f3eb2544b62389a79/8oU90du902ATtBPYCcLFK.jpeg
Ivan Hu
user
IvanHU
676d2c63310ca4eb397415ff
Huatong Song
admin_assigned
2024-12-30T21:02:13.538Z
66163dc8c7f45b3f893ff40b
/avatars/801043dac0caae90bbca8c9d3e2e203b.svg
Song Huatong
user
XXsongLALA
676d2c63310ca4eb39741600
Jia Deng
676d2c63310ca4eb39741601
Jiapeng Wang
admin_assigned
2024-12-30T21:01:54.842Z
61482b08c3ca9182d68debe9
/avatars/4319add48a092adfee3bc45930ac6d2a.svg
Jiapeng Wang
user
jpWang
676d2c63310ca4eb39741602
Jie Chen
claimed_verified
2025-03-10T12:46:37.345Z
651a29d566e78720a78317ec
https://cdn-avatars.huggingface.co/v1/production/uploads/651a29d566e78720a78317ec/8yTNvYeUs8cZo5GWvPPOJ.jpeg
Jie Chen
user
survivi
676d2c63310ca4eb39741603
Kun Zhou
676d2c63310ca4eb39741604
Yutao Zhu
admin_assigned
2024-12-30T21:01:41.536Z
625e62452a7279d3c77b5c38
https://cdn-avatars.huggingface.co/v1/production/uploads/625e62452a7279d3c77b5c38/zJINew6U4_Gup4WTobb-0.jpeg
Yutao Zhu
user
yutaozhu94
676d2c63310ca4eb39741605
Jinhao Jiang
admin_assigned
2024-12-30T21:01:35.432Z
61b8405b516a20acdf3b85ff
/avatars/3d2eae7c163a80b73260087b05a4230b.svg
Jinhao Jiang
user
Boru
676d2c63310ca4eb39741606
Zican Dong
admin_assigned
2024-12-30T21:01:29.767Z
65180fa499cf1a4e3913a31b
/avatars/a321b308336bb881b11b396e210226cf.svg
zican dong
user
cjgs20017
676d2c63310ca4eb39741607
Wayne Xin Zhao
676d2c63310ca4eb39741608
Ji-Rong Wen
admin_assigned
2024-12-30T21:01:14.755Z
64b8c89052b7353d8c6a1013
/avatars/cd59fffe81f6b07b4519540b8ff3d95f.svg
Ji-Rong Wen
user
jrwen
YuLan-Mini: An Open Data-efficient Language Model
Effective pre-training of large language models (LLMs) has been challenging
due to the immense resource demands and the complexity of the technical
processes involved. This paper presents a detailed technical report on
YuLan-Mini, a highly capable base model with 2.42B parameters that achieves
top-tier performance among models of similar parameter scale. Our pre-training
approach focuses on enhancing training efficacy through three key technical
contributions: an elaborate data pipeline combines data cleaning with data
schedule strategies, a robust optimization method to mitigate training
instability, and an effective annealing approach that incorporates targeted
data selection and long context training. Remarkably, YuLan-Mini, trained on
1.08T tokens, achieves performance comparable to industry-leading models that
require significantly more data. To facilitate reproduction, we release the
full details of the data composition for each training phase. Project details
can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini.
YuLan-Mini, a 2.42B parameter base model, achieves top-tier performance with efficient pre-training techniques, including a data pipeline with cleaning and scheduling, robust optimization, and annealing with targeted data selection.
data pipeline data cleaning data schedule strategies robust optimization annealing approach targeted data selection long context training