MiniCPM4: Ultra-Efficient LLMs on End Devices
This paper introduces MiniCPM4, a highly efficient large language model (LLM)
designed explicitly for end-side devices. We achieve this efficiency through
systematic innovation in four key dimensions: model architecture, training
data, training algorithms, and inference systems. Specifically, in terms of
model architecture, we propose InfLLM v2, a trainable sparse attention
mechanism that accelerates both prefilling and decoding phases for long-context
processing. Regarding training data, we propose UltraClean, an efficient and
accurate pre-training data filtering and generation strategy, and UltraChat v2,
a comprehensive supervised fine-tuning dataset. These datasets enable
satisfactory model performance to be achieved using just 8 trillion training
tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient
pre-training strategy search, and improve existing post-training methods by
introducing chunk-wise rollout for load-balanced reinforcement learning and
data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose
CPM.cu that integrates sparse attention, model quantization, and speculative
sampling to achieve efficient prefilling and decoding. To meet diverse
on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B
parameters, respectively. Sufficient evaluation results show that MiniCPM4
outperforms open-source models of similar size across multiple benchmarks,
highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B
demonstrates significant speed improvements over Qwen3-8B when processing long
sequences. Through further adaptation, MiniCPM4 successfully powers diverse
applications, including trustworthy survey generation and tool use with model
context protocol, clearly showcasing its broad usability.
6847924d3ec10bdd8ab4ddb9
MiniCPM Team
6847924d3ec10bdd8ab4ddba
Chaojun Xiao
claimed_verified
2025-06-10T11:00:03.612Z
608f6d72283d0a8d7be9d1f9
/avatars/7f499a37019359a3c488ba6cc11751fc.svg
Chaojun XIAO
user
xcjthu
6847924d3ec10bdd8ab4ddbb
Yuxuan Li
6847924d3ec10bdd8ab4ddbc
Xu Han
6847924d3ec10bdd8ab4ddbd
Yuzhuo Bai
6847924d3ec10bdd8ab4ddbe
Jie Cai
6847924d3ec10bdd8ab4ddbf
Haotian Chen
6847924d3ec10bdd8ab4ddc0
Wentong Chen
6847924d3ec10bdd8ab4ddc1
Xin Cong
6847924d3ec10bdd8ab4ddc2
Ganqu Cui
6847924d3ec10bdd8ab4ddc3
Ning Ding
6847924d3ec10bdd8ab4ddc4
Shengdan Fan
6847924d3ec10bdd8ab4ddc5
Yewei Fang
6847924d3ec10bdd8ab4ddc6
Zixuan Fu
6847924d3ec10bdd8ab4ddc7
Wenyu Guan
6847924d3ec10bdd8ab4ddc8
Yitong Guan
6847924d3ec10bdd8ab4ddc9
Junshao Guo
claimed_verified
2025-06-10T11:00:01.410Z
66add9c413ac672510f6cdba
https://cdn-avatars.huggingface.co/v1/production/uploads/66add9c413ac672510f6cdba/ubTWt8XN7aDdSihYAORah.png
guojunshao
user
guojunshaoyao
6847924d3ec10bdd8ab4ddca
Yufeng Han
6847924d3ec10bdd8ab4ddcb
Bingxiang He
6847924d3ec10bdd8ab4ddcc
Yuxiang Huang
6847924d3ec10bdd8ab4ddcd
Cunliang Kong
6847924d3ec10bdd8ab4ddce
Qiuzuo Li
6847924d3ec10bdd8ab4ddcf
Siyuan Li
6847924d3ec10bdd8ab4ddd0
Wenhao Li
6847924d3ec10bdd8ab4ddd1
Yanghao Li
6847924d3ec10bdd8ab4ddd2
Yishan Li
6847924d3ec10bdd8ab4ddd3
Zhen Li
6847924d3ec10bdd8ab4ddd4
Dan Liu
6847924d3ec10bdd8ab4ddd5
Biyuan Lin
6847924d3ec10bdd8ab4ddd6
Yankai Lin
6847924d3ec10bdd8ab4ddd7
Xiang Long
6847924d3ec10bdd8ab4ddd8
Quanyu Lu
6847924d3ec10bdd8ab4ddd9
Yaxi Lu
6847924d3ec10bdd8ab4ddda
Peiyan Luo
6847924d3ec10bdd8ab4dddb
Hongya Lyu
6847924d3ec10bdd8ab4dddc
Litu Ou
claimed_verified
2025-07-07T11:25:55.969Z
622f2feea32d46b4be9ed8c4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/NDeZQZQK5U-9m10yQwDVf.png
Litu Ou
user
learn3r
6847924d3ec10bdd8ab4dddd
Yinxu Pan
6847924d3ec10bdd8ab4ddde
Zekai Qu
6847924d3ec10bdd8ab4dddf
Qundong Shi
6847924d3ec10bdd8ab4dde0
Zijun Song
6847924d3ec10bdd8ab4dde1
Jiayuan Su
6847924d3ec10bdd8ab4dde2
Zhou Su
6847924d3ec10bdd8ab4dde3
Ao Sun
6847924d3ec10bdd8ab4dde4
Xianghui Sun
6847924d3ec10bdd8ab4dde5
Peijun Tang
claimed_verified
2025-06-11T08:36:11.840Z
642101b85bccaa424844df6e
/avatars/6015bbd6ff98fd5d61d690e0f8d60564.svg
tangpeijun
user
lumosity
6847924d3ec10bdd8ab4dde6
Fangzheng Wang
6847924d3ec10bdd8ab4dde7
Feng Wang
6847924d3ec10bdd8ab4dde8
Shuo Wang
6847924d3ec10bdd8ab4dde9
Yudong Wang
claimed_verified
2025-06-10T08:45:33.890Z
63be286fb3b8c44f8cecc16f
https://cdn-avatars.huggingface.co/v1/production/uploads/63be286fb3b8c44f8cecc16f/1CIkfEKoTnBYdYDSuQ8AT.jpeg
Yudong Wang
user
BigDong
6847924d3ec10bdd8ab4ddea
Yesai Wu
6847924d3ec10bdd8ab4ddeb
Zhenyu Xiao
6847924d3ec10bdd8ab4ddec
Jie Xie
6847924d3ec10bdd8ab4dded
Zihao Xie
6847924d3ec10bdd8ab4ddee
Yukun Yan
6847924d3ec10bdd8ab4ddef
Jiarui Yuan
6847924d3ec10bdd8ab4ddf0
Kaihuo Zhang
6847924d3ec10bdd8ab4ddf1
Lei Zhang
6847924d3ec10bdd8ab4ddf2
Linyue Zhang
6847924d3ec10bdd8ab4ddf3
Xueren Zhang
6847924d3ec10bdd8ab4ddf4
Yudi Zhang
6847924d3ec10bdd8ab4ddf5
Hengyu Zhao
6847924d3ec10bdd8ab4ddf6
Weilin Zhao
6847924d3ec10bdd8ab4ddf7
Weilun Zhao
6847924d3ec10bdd8ab4ddf8
Yuanqian Zhao
6847924d3ec10bdd8ab4ddf9
Zhi Zheng
6847924d3ec10bdd8ab4ddfa
Ge Zhou
6847924d3ec10bdd8ab4ddfb
Jie Zhou
6847924d3ec10bdd8ab4ddfc
Wei Zhou
6847924d3ec10bdd8ab4ddfd
Zihan Zhou
6847924d3ec10bdd8ab4ddfe
Zixuan Zhou
6847924d3ec10bdd8ab4ddff
Zhiyuan Liu
6847924d3ec10bdd8ab4de00
Guoyang Zeng
6847924d3ec10bdd8ab4de01
Chao Jia
6847924d3ec10bdd8ab4de02
Dahai Li
6847924d3ec10bdd8ab4de03
Maosong Sun
MiniCPM4: Ultra-Efficient LLMs on End Devices
This paper introduces MiniCPM4, a highly efficient large language model (LLM)
designed explicitly for end-side devices. We achieve this efficiency through
systematic innovation in four key dimensions: model architecture, training
data, training algorithms, and inference systems. Specifically, in terms of
model architecture, we propose InfLLM v2, a trainable sparse attention
mechanism that accelerates both prefilling and decoding phases for long-context
processing. Regarding training data, we propose UltraClean, an efficient and
accurate pre-training data filtering and generation strategy, and UltraChat v2,
a comprehensive supervised fine-tuning dataset. These datasets enable
satisfactory model performance to be achieved using just 8 trillion training
tokens. Regarding training algorithms, we propose ModelTunnel v2 for efficient
pre-training strategy search, and improve existing post-training methods by
introducing chunk-wise rollout for load-balanced reinforcement learning and
data-efficient tenary LLM, BitCPM. Regarding inference systems, we propose
CPM.cu that integrates sparse attention, model quantization, and speculative
sampling to achieve efficient prefilling and decoding. To meet diverse
on-device requirements, MiniCPM4 is available in two versions, with 0.5B and 8B
parameters, respectively. Sufficient evaluation results show that MiniCPM4
outperforms open-source models of similar size across multiple benchmarks,
highlighting both its efficiency and effectiveness. Notably, MiniCPM4-8B
demonstrates significant speed improvements over Qwen3-8B when processing long
sequences. Through further adaptation, MiniCPM4 successfully powers diverse
applications, including trustworthy survey generation and tool use with model
context protocol, clearly showcasing its broad usability.
MiniCPM4, a highly efficient large language model for end-side devices, achieves superior performance using innovations in sparse attention, pre-training datasets, training algorithms, and inference systems.
InfLLM v2 sparse attention mechanism UltraClean UltraChat v2 prefilling decoding long-context processing ModelTunnel v2 chunk-wise rollout data-efficient tenary LLM BitCPM CPM.cu model quantization speculative sampling