MiniCPM-V: A GPT-4V Level MLLM on Your Phone
The recent surge of Multimodal Large Language Models (MLLMs) has
fundamentally reshaped the landscape of AI research and industry, shedding
light on a promising path toward the next AI milestone. However, significant
challenges remain preventing MLLMs from being practical in real-world
applications. The most notable challenge comes from the huge cost of running an
MLLM with a massive number of parameters and extensive computation. As a
result, most MLLMs need to be deployed on high-performing cloud servers, which
greatly limits their application scopes such as mobile, offline,
energy-sensitive, and privacy-protective scenarios. In this work, we present
MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
integrating the latest MLLM techniques in architecture, pretraining and
alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
OCR capability and 1.8M pixel high-resolution image perception at any aspect
ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
support for 30+ languages, and (5) efficient deployment on mobile phones. More
importantly, MiniCPM-V can be viewed as a representative example of a promising
trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
are rapidly decreasing, along with the fast growth of end-side computation
capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
are becoming increasingly possible, unlocking a wider spectrum of real-world AI
applications in the near future.
66b1a282bb6fcf48f1acf552
Yuan Yao
66b1a282bb6fcf48f1acf553
Tianyu Yu
admin_assigned
2024-08-06T08:01:00.844Z
64abc4aa6cadc7aca585dddf
/avatars/736afea979cd0021c7a37f68731524ea.svg
Tianyu Yu
user
Yirany
66b1a282bb6fcf48f1acf554
Ao Zhang
admin_assigned
2024-08-06T08:01:16.419Z
649a9133225ce03f1c82e0b2
/avatars/f3702a46a0a310d0292dc690794e7cc9.svg
Ao Zhang
user
AoZhang
66b1a282bb6fcf48f1acf555
Chongyi Wang
claimed_verified
2025-01-14T08:30:30.969Z
6350fa8385bdb764f6a9aa82
https://cdn-avatars.huggingface.co/v1/production/uploads/6350fa8385bdb764f6a9aa82/QBp92D7x9_XImYTigBHsu.jpeg
chongyi
user
yuzaa
66b1a282bb6fcf48f1acf556
Junbo Cui
claimed_verified
2024-08-06T13:25:32.958Z
63f706dfe94ed998c463ed66
https://cdn-avatars.huggingface.co/v1/production/uploads/63f706dfe94ed998c463ed66/tAGil2qiFNev6CfEEDseV.png
Cuiunbo
user
Cuiunbo
66b1a282bb6fcf48f1acf557
Hongji Zhu
admin_assigned
2024-08-06T08:54:16.536Z
634e5b942b8c811f3cbd7052
/avatars/1befeda87e8332f40fbd23e334d885cc.svg
Hongji Zhu
user
finalf0
66b1a282bb6fcf48f1acf558
Tianchi Cai
admin_assigned
2024-08-06T08:54:23.184Z
65164295d467ffdc05278d1f
/avatars/37a7fe0088e5b9eca5479a4538242073.svg
Cai
user
tianchicai
66b1a282bb6fcf48f1acf559
Haoyu Li
claimed_verified
2024-08-06T15:25:28.188Z
64c760563e30498ccad7d599
https://cdn-avatars.huggingface.co/v1/production/uploads/64c760563e30498ccad7d599/35G2pXRMXEKyOAglIzoXx.jpeg
Haoyu Li
user
trainfanlhy
66b1a282bb6fcf48f1acf55a
Weilin Zhao
admin_assigned
2024-08-06T08:54:44.751Z
64b8ff3d95bd42c770878042
/avatars/564a4dccdf9e5b813a99979b0ef58183.svg
Weilin Zhao
user
Achazwl
66b1a282bb6fcf48f1acf55b
Zhihui He
admin_assigned
2024-08-06T08:54:51.371Z
659fa38f83abded48e3fd315
/avatars/0daeaafc96306e061db411fec62ce40a.svg
Zhihui He
user
HwwwH
66b1a282bb6fcf48f1acf55c
Qianyu Chen
admin_assigned
2024-08-06T08:54:57.168Z
62518fa339d59a914cd778a5
/avatars/c273176cc9e180859571f65fb39d99f3.svg
qianyu chen
user
qianyuchen
66b1a282bb6fcf48f1acf55d
Huarong Zhou
66b1a282bb6fcf48f1acf55e
Zhensheng Zou
66b1a282bb6fcf48f1acf55f
Haoye Zhang
admin_assigned
2024-08-06T08:55:32.240Z
6566e0c493e30c8a60048eb3
/avatars/78a021f1b205dabef3d060d1edc33bb7.svg
Haoye Zhang
user
HaoyeZhang
66b1a282bb6fcf48f1acf560
Shengding Hu
admin_assigned
2024-08-06T08:55:38.607Z
6041bec6c15e823a685f2afc
/avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
Shengding Hu
user
ShengdingHu
66b1a282bb6fcf48f1acf561
Zhi Zheng
claimed_verified
2024-10-13T20:18:55.651Z
62d0d2f1c1d35087f508fbea
https://cdn-avatars.huggingface.co/v1/production/uploads/1676873251372-62d0d2f1c1d35087f508fbea.jpeg
Zhi Zheng
user
neoz
66b1a282bb6fcf48f1acf562
Jie Zhou
66b1a282bb6fcf48f1acf563
Jie Cai
66b1a282bb6fcf48f1acf564
Xu Han
66b1a282bb6fcf48f1acf565
Guoyang Zeng
admin_assigned
2024-08-06T08:57:34.209Z
60cd679ef3c0385f86cb07c2
https://cdn-avatars.huggingface.co/v1/production/uploads/1624074137194-noauth.jpeg
Guoyang Zeng
user
losfen
66b1a282bb6fcf48f1acf566
Dahai Li
admin_assigned
2024-08-06T08:57:40.081Z
6457c0bd70b133710996e99b
/avatars/eefbdbcb80d832360509d301bf09f845.svg
dahai li
user
dahai2023
66b1a282bb6fcf48f1acf567
Zhiyuan Liu
admin_assigned
2024-08-06T08:57:49.338Z
6472f1f7485a7c8e1bcf507b
/avatars/79dca137583875191ea883b193b630b7.svg
Zhiyuan Liu
user
zibuyu9
66b1a282bb6fcf48f1acf568
Maosong Sun
MiniCPM-V: A GPT-4V Level MLLM on Your Phone
The recent surge of Multimodal Large Language Models (MLLMs) has
fundamentally reshaped the landscape of AI research and industry, shedding
light on a promising path toward the next AI milestone. However, significant
challenges remain preventing MLLMs from being practical in real-world
applications. The most notable challenge comes from the huge cost of running an
MLLM with a massive number of parameters and extensive computation. As a
result, most MLLMs need to be deployed on high-performing cloud servers, which
greatly limits their application scopes such as mobile, offline,
energy-sensitive, and privacy-protective scenarios. In this work, we present
MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
integrating the latest MLLM techniques in architecture, pretraining and
alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
OCR capability and 1.8M pixel high-resolution image perception at any aspect
ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
support for 30+ languages, and (5) efficient deployment on mobile phones. More
importantly, MiniCPM-V can be viewed as a representative example of a promising
trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
are rapidly decreasing, along with the fast growth of end-side computation
capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
are becoming increasingly possible, unlocking a wider spectrum of real-world AI
applications in the near future.
MiniCPM-V presents a series of efficient Multimodal Large Language Models optimized for end-side deployment, offering high performance and practical usability compared to larger models.
Multimodal Large Language Models MLLMs end-side devices architecture pretraining alignment OpenCompass OCR image perception hallucination multilingual support