MiniCPM: Unveiling the Potential of Small Language Models with Scalable
  Training Strategies
The burgeoning interest in developing Large Language Models (LLMs) with up to
trillion parameters has been met with concerns regarding resource efficiency
and practical expense, particularly given the immense cost of experimentation.
This scenario underscores the importance of exploring the potential of Small
Language Models (SLMs) as a resource-efficient alternative. In this context, we
introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter
variants, not only excel in their respective categories but also demonstrate
capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach
exhibits scalability in both model and data dimensions for future LLM research.
Regarding model scaling, we employ extensive model wind tunnel experiments for
stable and optimal scaling. For data scaling, we introduce a
Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to
continuous training and domain adaptation. We present an in-depth analysis of
the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we
are now able to efficiently study data-model scaling law without extensive
retraining experiments on both axes of model and data, from which we derive the
much higher compute optimal data-model ratio than Chinchilla Optimal.
Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE
and MiniCPM-128K, whose excellent performance further cementing MiniCPM's
foundation in diverse SLM applications. MiniCPM models are available publicly
at https://github.com/OpenBMB/MiniCPM .
6615f84fa5849fc8a2780270
Shengding Hu
extracted_pending
2024-04-10T02:24:17.528Z
6041bec6c15e823a685f2afc
/avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
Shengding Hu
user
ShengdingHu
6615f84fa5849fc8a2780271
Yuge Tu
admin_assigned
2024-04-22T14:16:41.418Z
637f3220e0d5fd3199579458
https://cdn-avatars.huggingface.co/v1/production/uploads/637f3220e0d5fd3199579458/uaPvnmGqRcScz9RgoUcVY.png
Yuge Tu
user
tuyuge
6615f84fa5849fc8a2780272
Xu Han
6615f84fa5849fc8a2780273
Chaoqun He
claimed_verified
2025-07-28T08:48:27.879Z
64ca3bec2e592905f9c66385
/avatars/02f7e38072e37d4a882c22dc6aac139c.svg
Chaoqun He
user
Hothan
6615f84fa5849fc8a2780274
Ganqu Cui
admin_assigned
2024-04-22T14:16:55.848Z
650eba9555dc1e841746f132
/avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
Ganqu Cui
user
ganqu
6615f84fa5849fc8a2780275
Xiang Long
6615f84fa5849fc8a2780276
Zhi Zheng
claimed_verified
2024-10-13T20:19:53.306Z
62d0d2f1c1d35087f508fbea
https://cdn-avatars.huggingface.co/v1/production/uploads/1676873251372-62d0d2f1c1d35087f508fbea.jpeg
Zhi Zheng
user
neoz
6615f84fa5849fc8a2780277
Yewei Fang
6615f84fa5849fc8a2780278
Yuxiang Huang
claimed_verified
2025-03-05T10:13:10.521Z
65394f942ddab2beff1cc967
/avatars/bdd5b47618758f6de080a51834a17655.svg
Shawn Huang
user
hyx21
6615f84fa5849fc8a2780279
Weilin Zhao
admin_assigned
2024-04-22T14:17:28.646Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
6615f84fa5849fc8a278027a
Xinrong Zhang
admin_assigned
2024-07-30T14:54:29.694Z
6316eec27b0ee0136e5e8132
/avatars/a08f7cbaf155d8cef3dd6197a21890a5.svg
xinrongzhang
user
xinrongzhang2022
6615f84fa5849fc8a278027b
Zheng Leng Thai
6615f84fa5849fc8a278027c
Kaihuo Zhang
6615f84fa5849fc8a278027d
Chongyi Wang
claimed_verified
2025-01-14T08:30:42.523Z
6350fa8385bdb764f6a9aa82
https://cdn-avatars.huggingface.co/v1/production/uploads/6350fa8385bdb764f6a9aa82/QBp92D7x9_XImYTigBHsu.jpeg
chongyi
user
yuzaa
6615f84fa5849fc8a278027e
Yuan Yao
6615f84fa5849fc8a278027f
Chenyang Zhao
6615f84fa5849fc8a2780280
Jie Zhou
6615f84fa5849fc8a2780281
Jie Cai
6615f84fa5849fc8a2780282
Zhongwu Zhai
6615f84fa5849fc8a2780283
Ning Ding
admin_assigned
2024-07-30T14:55:26.402Z
60cf4bcb1ce3775ebb86e5d5
/avatars/12bcd18d215abf91f297f93007733148.svg
Ning Ding
user
stingning
6615f84fa5849fc8a2780284
Chao Jia
6615f84fa5849fc8a2780285
Guoyang Zeng
admin_assigned
2024-07-30T14:54:53.649Z
60cd679ef3c0385f86cb07c2
https://cdn-avatars.huggingface.co/v1/production/uploads/1624074137194-noauth.jpeg
Guoyang Zeng
user
losfen
6615f84fa5849fc8a2780286
Dahai Li
admin_assigned
2024-07-30T14:56:03.346Z
6457c0bd70b133710996e99b
/avatars/eefbdbcb80d832360509d301bf09f845.svg
dahai li
user
dahai2023
6615f84fa5849fc8a2780287
Zhiyuan Liu
admin_assigned
2024-07-30T14:56:22.812Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
6615f84fa5849fc8a2780288
Maosong Sun
MiniCPM: Unveiling the Potential of Small Language Models with Scalable
  Training Strategies
The burgeoning interest in developing Large Language Models (LLMs) with up to
trillion parameters has been met with concerns regarding resource efficiency
and practical expense, particularly given the immense cost of experimentation.
This scenario underscores the importance of exploring the potential of Small
Language Models (SLMs) as a resource-efficient alternative. In this context, we
introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter
variants, not only excel in their respective categories but also demonstrate
capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach
exhibits scalability in both model and data dimensions for future LLM research.
Regarding model scaling, we employ extensive model wind tunnel experiments for
stable and optimal scaling. For data scaling, we introduce a
Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to
continuous training and domain adaptation. We present an in-depth analysis of
the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we
are now able to efficiently study data-model scaling law without extensive
retraining experiments on both axes of model and data, from which we derive the
much higher compute optimal data-model ratio than Chinchilla Optimal.
Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE
and MiniCPM-128K, whose excellent performance further cementing MiniCPM's
foundation in diverse SLM applications. MiniCPM models are available publicly
at https://github.com/OpenBMB/MiniCPM .
MiniCPM, a family of small language models, demonstrates performance comparable to large models while introducing efficient scaling techniques and learning rate management.
Large Language Models Small Language Models model wind tunnel Warmup-Stable-Decay learning rate scheduler MiniCPM-DPO MiniCPM-MoE MiniCPM-128K data-model scaling law Chinchilla Optimal