title: MiniCPM-V: A GPT-4V Level MLLM on Your Phone
summary: The recent surge of Multimodal Large Language Models (MLLMs) has
  fundamentally reshaped the landscape of AI research and industry, shedding
  light on a promising path toward the next AI milestone. However, significant
  challenges remain preventing MLLMs from being practical in real-world
  applications. The most notable challenge comes from the huge cost of running an
  MLLM with a massive number of parameters and extensive computation. As a
  result, most MLLMs need to be deployed on high-performing cloud servers, which
  greatly limits their application scopes such as mobile, offline,
  energy-sensitive, and privacy-protective scenarios. In this work, we present
  MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
  integrating the latest MLLM techniques in architecture, pretraining and
  alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
  Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
  OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
  OCR capability and 1.8M pixel high-resolution image perception at any aspect
  ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
  support for 30+ languages, and (5) efficient deployment on mobile phones. More
  importantly, MiniCPM-V can be viewed as a representative example of a promising
  trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
  are rapidly decreasing, along with the fast growth of end-side computation
  capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
  are becoming increasingly possible, unlocking a wider spectrum of real-world AI
  applications in the near future.
paper_authors._id: 66b1a282bb6fcf48f1acf552
paper_authors.name: Yuan Yao
paper_authors._id: 66b1a282bb6fcf48f1acf553
paper_authors.name: Tianyu Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:01:00.844Z
paper_authors._id: 64abc4aa6cadc7aca585dddf
paper_authors.avatarUrl: /avatars/736afea979cd0021c7a37f68731524ea.svg
paper_authors.fullname: Tianyu Yu
paper_authors.type: user
paper_authors.user: Yirany
paper_authors._id: 66b1a282bb6fcf48f1acf554
paper_authors.name: Ao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:01:16.419Z
paper_authors._id: 649a9133225ce03f1c82e0b2
paper_authors.avatarUrl: /avatars/f3702a46a0a310d0292dc690794e7cc9.svg
paper_authors.fullname: Ao Zhang
paper_authors.type: user
paper_authors.user: AoZhang
paper_authors._id: 66b1a282bb6fcf48f1acf555
paper_authors.name: Chongyi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-14T08:30:30.969Z
paper_authors._id: 6350fa8385bdb764f6a9aa82
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6350fa8385bdb764f6a9aa82/QBp92D7x9_XImYTigBHsu.jpeg
paper_authors.fullname: chongyi
paper_authors.type: user
paper_authors.user: yuzaa
paper_authors._id: 66b1a282bb6fcf48f1acf556
paper_authors.name: Junbo Cui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-06T13:25:32.958Z
paper_authors._id: 63f706dfe94ed998c463ed66
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63f706dfe94ed998c463ed66/tAGil2qiFNev6CfEEDseV.png
paper_authors.fullname: Cuiunbo
paper_authors.type: user
paper_authors.user: Cuiunbo
paper_authors._id: 66b1a282bb6fcf48f1acf557
paper_authors.name: Hongji Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:54:16.536Z
paper_authors._id: 634e5b942b8c811f3cbd7052
paper_authors.avatarUrl: /avatars/1befeda87e8332f40fbd23e334d885cc.svg
paper_authors.fullname: Hongji Zhu
paper_authors.type: user
paper_authors.user: finalf0
paper_authors._id: 66b1a282bb6fcf48f1acf558
paper_authors.name: Tianchi Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:54:23.184Z
paper_authors._id: 65164295d467ffdc05278d1f
paper_authors.avatarUrl: /avatars/37a7fe0088e5b9eca5479a4538242073.svg
paper_authors.fullname: Cai
paper_authors.type: user
paper_authors.user: tianchicai
paper_authors._id: 66b1a282bb6fcf48f1acf559
paper_authors.name: Haoyu Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-06T15:25:28.188Z
paper_authors._id: 64c760563e30498ccad7d599
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c760563e30498ccad7d599/35G2pXRMXEKyOAglIzoXx.jpeg
paper_authors.fullname: Haoyu Li
paper_authors.type: user
paper_authors.user: trainfanlhy
paper_authors._id: 66b1a282bb6fcf48f1acf55a
paper_authors.name: Weilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:54:44.751Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 66b1a282bb6fcf48f1acf55b
paper_authors.name: Zhihui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:54:51.371Z
paper_authors._id: 659fa38f83abded48e3fd315
paper_authors.avatarUrl: /avatars/0daeaafc96306e061db411fec62ce40a.svg
paper_authors.fullname: Zhihui He
paper_authors.type: user
paper_authors.user: HwwwH
paper_authors._id: 66b1a282bb6fcf48f1acf55c
paper_authors.name: Qianyu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:54:57.168Z
paper_authors._id: 62518fa339d59a914cd778a5
paper_authors.avatarUrl: /avatars/c273176cc9e180859571f65fb39d99f3.svg
paper_authors.fullname: qianyu chen
paper_authors.type: user
paper_authors.user: qianyuchen
paper_authors._id: 66b1a282bb6fcf48f1acf55d
paper_authors.name: Huarong Zhou
paper_authors._id: 66b1a282bb6fcf48f1acf55e
paper_authors.name: Zhensheng Zou
paper_authors._id: 66b1a282bb6fcf48f1acf55f
paper_authors.name: Haoye Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:55:32.240Z
paper_authors._id: 6566e0c493e30c8a60048eb3
paper_authors.avatarUrl: /avatars/78a021f1b205dabef3d060d1edc33bb7.svg
paper_authors.fullname: Haoye Zhang
paper_authors.type: user
paper_authors.user: HaoyeZhang
paper_authors._id: 66b1a282bb6fcf48f1acf560
paper_authors.name: Shengding Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:55:38.607Z
paper_authors._id: 6041bec6c15e823a685f2afc
paper_authors.avatarUrl: /avatars/7d09fbe771d3c3312ac7eb6ef83ce053.svg
paper_authors.fullname: Shengding Hu
paper_authors.type: user
paper_authors.user: ShengdingHu
paper_authors._id: 66b1a282bb6fcf48f1acf561
paper_authors.name: Zhi Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:18:55.651Z
paper_authors._id: 62d0d2f1c1d35087f508fbea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676873251372-62d0d2f1c1d35087f508fbea.jpeg
paper_authors.fullname: Zhi Zheng
paper_authors.type: user
paper_authors.user: neoz
paper_authors._id: 66b1a282bb6fcf48f1acf562
paper_authors.name: Jie Zhou
paper_authors._id: 66b1a282bb6fcf48f1acf563
paper_authors.name: Jie Cai
paper_authors._id: 66b1a282bb6fcf48f1acf564
paper_authors.name: Xu Han
paper_authors._id: 66b1a282bb6fcf48f1acf565
paper_authors.name: Guoyang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:57:34.209Z
paper_authors._id: 60cd679ef3c0385f86cb07c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624074137194-noauth.jpeg
paper_authors.fullname: Guoyang Zeng
paper_authors.type: user
paper_authors.user: losfen
paper_authors._id: 66b1a282bb6fcf48f1acf566
paper_authors.name: Dahai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:57:40.081Z
paper_authors._id: 6457c0bd70b133710996e99b
paper_authors.avatarUrl: /avatars/eefbdbcb80d832360509d301bf09f845.svg
paper_authors.fullname: dahai li
paper_authors.type: user
paper_authors.user: dahai2023
paper_authors._id: 66b1a282bb6fcf48f1acf567
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-06T08:57:49.338Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 66b1a282bb6fcf48f1acf568
paper_authors.name: Maosong Sun
paper_title: MiniCPM-V: A GPT-4V Level MLLM on Your Phone
paper_summary: The recent surge of Multimodal Large Language Models (MLLMs) has
  fundamentally reshaped the landscape of AI research and industry, shedding
  light on a promising path toward the next AI milestone. However, significant
  challenges remain preventing MLLMs from being practical in real-world
  applications. The most notable challenge comes from the huge cost of running an
  MLLM with a massive number of parameters and extensive computation. As a
  result, most MLLMs need to be deployed on high-performing cloud servers, which
  greatly limits their application scopes such as mobile, offline,
  energy-sensitive, and privacy-protective scenarios. In this work, we present
  MiniCPM-V, a series of efficient MLLMs deployable on end-side devices. By
  integrating the latest MLLM techniques in architecture, pretraining and
  alignment, the latest MiniCPM-Llama3-V 2.5 has several notable features: (1)
  Strong performance, outperforming GPT-4V-1106, Gemini Pro and Claude 3 on
  OpenCompass, a comprehensive evaluation over 11 popular benchmarks, (2) strong
  OCR capability and 1.8M pixel high-resolution image perception at any aspect
  ratio, (3) trustworthy behavior with low hallucination rates, (4) multilingual
  support for 30+ languages, and (5) efficient deployment on mobile phones. More
  importantly, MiniCPM-V can be viewed as a representative example of a promising
  trend: The model sizes for achieving usable (e.g., GPT-4V) level performance
  are rapidly decreasing, along with the fast growth of end-side computation
  capacity. This jointly shows that GPT-4V level MLLMs deployed on end devices
  are becoming increasingly possible, unlocking a wider spectrum of real-world AI
  applications in the near future.
paper_ai_summary: MiniCPM-V presents a series of efficient Multimodal Large Language Models optimized for end-side deployment, offering high performance and practical usability compared to larger models.
paper_ai_keywords: Multimodal Large Language Models MLLMs end-side devices architecture pretraining alignment OpenCompass OCR image perception hallucination multilingual support