title: MiniCPM: Unveiling the Potential of Small Language Models with Scalable
    Training Strategies
summary: The burgeoning interest in developing Large Language Models (LLMs) with up to
  trillion parameters has been met with concerns regarding resource efficiency
  and practical expense, particularly given the immense cost of experimentation.
  This scenario underscores the importance of exploring the potential of Small
  Language Models (SLMs) as a resource-efficient alternative. In this context, we
  introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter
  variants, not only excel in their respective categories but also demonstrate
  capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach
  exhibits scalability in both model and data dimensions for future LLM research.
  Regarding model scaling, we employ extensive model wind tunnel experiments for
  stable and optimal scaling. For data scaling, we introduce a
  Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to
  continuous training and domain adaptation. We present an in-depth analysis of
  the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we
  are now able to efficiently study data-model scaling law without extensive
  retraining experiments on both axes of model and data, from which we derive the
  much higher compute optimal data-model ratio than Chinchilla Optimal.
  Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE
  and MiniCPM-128K, whose excellent performance further cementing MiniCPM's
  foundation in diverse SLM applications. MiniCPM models are available publicly
  at https://github.com/OpenBMB/MiniCPM .
paper_authors._id: 6615f84fa5849fc8a2780270
paper_authors.name: Shengding Hu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-04-10T02:24:17.528Z
paper_authors._id: 6041bec6c15e823a685f2afc
paper_authors.avatarUrl: /avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
paper_authors.fullname: Shengding Hu
paper_authors.type: user
paper_authors.user: ShengdingHu
paper_authors._id: 6615f84fa5849fc8a2780271
paper_authors.name: Yuge Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:16:41.418Z
paper_authors._id: 637f3220e0d5fd3199579458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f3220e0d5fd3199579458/uaPvnmGqRcScz9RgoUcVY.png
paper_authors.fullname: Yuge Tu
paper_authors.type: user
paper_authors.user: tuyuge
paper_authors._id: 6615f84fa5849fc8a2780272
paper_authors.name: Xu Han
paper_authors._id: 6615f84fa5849fc8a2780273
paper_authors.name: Chaoqun He
paper_authors._id: 6615f84fa5849fc8a2780274
paper_authors.name: Ganqu Cui
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:16:55.848Z
paper_authors._id: 650eba9555dc1e841746f132
paper_authors.avatarUrl: /avatars/af6f5ee78f161d25ec0afc45d2def8eb.svg
paper_authors.fullname: Ganqu Cui
paper_authors.type: user
paper_authors.user: ganqu
paper_authors._id: 6615f84fa5849fc8a2780275
paper_authors.name: Xiang Long
paper_authors._id: 6615f84fa5849fc8a2780276
paper_authors.name: Zhi Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:19:53.306Z
paper_authors._id: 62d0d2f1c1d35087f508fbea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676873251372-62d0d2f1c1d35087f508fbea.jpeg
paper_authors.fullname: Zhi Zheng
paper_authors.type: user
paper_authors.user: neoz
paper_authors._id: 6615f84fa5849fc8a2780277
paper_authors.name: Yewei Fang
paper_authors._id: 6615f84fa5849fc8a2780278
paper_authors.name: Yuxiang Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-05T10:13:10.521Z
paper_authors._id: 65394f942ddab2beff1cc967
paper_authors.avatarUrl: /avatars/bdd5b47618758f6de080a51834a17655.svg
paper_authors.fullname: Shawn Huang
paper_authors.type: user
paper_authors.user: hyx21
paper_authors._id: 6615f84fa5849fc8a2780279
paper_authors.name: Weilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-22T14:17:28.646Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 6615f84fa5849fc8a278027a
paper_authors.name: Xinrong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T14:54:29.694Z
paper_authors._id: 6316eec27b0ee0136e5e8132
paper_authors.avatarUrl: /avatars/a08f7cbaf155d8cef3dd6197a21890a5.svg
paper_authors.fullname: xinrongzhang
paper_authors.type: user
paper_authors.user: xinrongzhang2022
paper_authors._id: 6615f84fa5849fc8a278027b
paper_authors.name: Zheng Leng Thai
paper_authors._id: 6615f84fa5849fc8a278027c
paper_authors.name: Kaihuo Zhang
paper_authors._id: 6615f84fa5849fc8a278027d
paper_authors.name: Chongyi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:30:42.523Z
paper_authors._id: 6350fa8385bdb764f6a9aa82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6350fa8385bdb764f6a9aa82/QBp92D7x9_XImYTigBHsu.jpeg
paper_authors.fullname: chongyi
paper_authors.type: user
paper_authors.user: yuzaa
paper_authors._id: 6615f84fa5849fc8a278027e
paper_authors.name: Yuan Yao
paper_authors._id: 6615f84fa5849fc8a278027f
paper_authors.name: Chenyang Zhao
paper_authors._id: 6615f84fa5849fc8a2780280
paper_authors.name: Jie Zhou
paper_authors._id: 6615f84fa5849fc8a2780281
paper_authors.name: Jie Cai
paper_authors._id: 6615f84fa5849fc8a2780282
paper_authors.name: Zhongwu Zhai
paper_authors._id: 6615f84fa5849fc8a2780283
paper_authors.name: Ning Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T14:55:26.402Z
paper_authors._id: 60cf4bcb1ce3775ebb86e5d5
paper_authors.avatarUrl: /avatars/12bcd18d215abf91f297f93007733148.svg
paper_authors.fullname: Ning Ding
paper_authors.type: user
paper_authors.user: stingning
paper_authors._id: 6615f84fa5849fc8a2780284
paper_authors.name: Chao Jia
paper_authors._id: 6615f84fa5849fc8a2780285
paper_authors.name: Guoyang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T14:54:53.649Z
paper_authors._id: 60cd679ef3c0385f86cb07c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624074137194-noauth.jpeg
paper_authors.fullname: Guoyang Zeng
paper_authors.type: user
paper_authors.user: losfen
paper_authors._id: 6615f84fa5849fc8a2780286
paper_authors.name: Dahai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T14:56:03.346Z
paper_authors._id: 6457c0bd70b133710996e99b
paper_authors.avatarUrl: /avatars/eefbdbcb80d832360509d301bf09f845.svg
paper_authors.fullname: dahai li
paper_authors.type: user
paper_authors.user: dahai2023
paper_authors._id: 6615f84fa5849fc8a2780287
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-30T14:56:22.812Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 6615f84fa5849fc8a2780288
paper_authors.name: Maosong Sun
paper_title: MiniCPM: Unveiling the Potential of Small Language Models with Scalable
    Training Strategies
paper_summary: The burgeoning interest in developing Large Language Models (LLMs) with up to
  trillion parameters has been met with concerns regarding resource efficiency
  and practical expense, particularly given the immense cost of experimentation.
  This scenario underscores the importance of exploring the potential of Small
  Language Models (SLMs) as a resource-efficient alternative. In this context, we
  introduce MiniCPM, specifically the 1.2B and 2.4B non-embedding parameter
  variants, not only excel in their respective categories but also demonstrate
  capabilities on par with 7B-13B LLMs. While focusing on SLMs, our approach
  exhibits scalability in both model and data dimensions for future LLM research.
  Regarding model scaling, we employ extensive model wind tunnel experiments for
  stable and optimal scaling. For data scaling, we introduce a
  Warmup-Stable-Decay (WSD) learning rate scheduler (LRS), conducive to
  continuous training and domain adaptation. We present an in-depth analysis of
  the intriguing training dynamics that occurred in the WSD LRS. With WSD LRS, we
  are now able to efficiently study data-model scaling law without extensive
  retraining experiments on both axes of model and data, from which we derive the
  much higher compute optimal data-model ratio than Chinchilla Optimal.
  Additionally, we introduce MiniCPM family, including MiniCPM-DPO, MiniCPM-MoE
  and MiniCPM-128K, whose excellent performance further cementing MiniCPM's
  foundation in diverse SLM applications. MiniCPM models are available publicly
  at https://github.com/OpenBMB/MiniCPM .
paper_ai_keywords: MiniCPM Small Language Models (SLMs) parameter-efficient model wind tunnel experiments Warmup-Stable-Decay (WSD) learning rate scheduler (LRS) continuous training data scaling law compute optimal data-model ratio Chinchilla Optimal MiniCPM-DPO MiniCPM-MoE MiniCPM-128K