Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal
  Large Language Models
This paper focuses on monolithic Multimodal Large Language Models (MLLMs),
which integrate visual encoding and language decoding into a single model.
Existing structures and pre-training strategies for monolithic MLLMs often
suffer from unstable optimization and catastrophic forgetting. To address these
challenges, our key idea is to embed a new visual parameter space into a
pre-trained LLM, enabling stable learning of visual knowledge from noisy data
via delta tuning. Based on this principle, we first introduce Mono-InternVL, an
advanced monolithic MLLM that incorporates a set of visual experts through a
multimodal mixture-of-experts architecture. In addition, we design an
innovative Endogenous Visual Pre-training (EViP) for Mono-InternVL to maximize
its visual capabilities via progressive learning. Mono-InternVL achieves
competitive performance against existing MLLMs but also leads to relatively
expensive data cost. Therefore, we further present Mono-InternVL-1.5, a cheaper
and stronger monolithic MLLM equipped with an improved EViP (EViP++). EViP++
introduces additional visual attention experts to Mono-InternVL-1.5 and
re-organizes the pre-training process in an efficient manner. During inference,
it includes a fused CUDA kernel to speed up its MoE operations. With these
designs, Mono-InternVL-1.5 significantly reduces training and inference costs,
while still maintaining competitive performance with Mono-InternVL. To evaluate
our approach, we conduct extensive experiments across 15 benchmarks. Results
demonstrate that Mono-InternVL outperforms existing monolithic MLLMs on 12 out
of 15 benchmarks, e.g., +114-point improvement over Emu3 on OCRBench. Compared
to its modular counterpart, i.e., InternVL-1.5, Mono-InternVL-1.5 achieves
similar multimodal performance while reducing first-token latency by up to 69%.
Code and models are released at https://github.com/OpenGVLab/Mono-InternVL.
687dd61e2e8db0930be6f107
Gen Luo
687dd61e2e8db0930be6f108
Wenhan Dou
admin_assigned
2025-07-21T15:59:10.836Z
66efe658de163a536aa84178
/avatars/de547ed48ca37694e08ee926aea03d9f.svg
dou wenhan
user
douwh
687dd61e2e8db0930be6f109
Wenhao Li
687dd61e2e8db0930be6f10a
Zhaokai Wang
claimed_verified
2025-07-21T08:29:25.907Z
665d4b515fdfe8f923e347a7
/avatars/d114b24c02dadfca0a8aee104755a8ec.svg
Zhaokai Wang
user
wzk1015
687dd61e2e8db0930be6f10b
Xue Yang
687dd61e2e8db0930be6f10c
Changyao Tian
admin_assigned
2025-07-21T15:58:59.906Z
64b7475efa7eabaae5f7ba94
https://cdn-avatars.huggingface.co/v1/production/uploads/64b7475efa7eabaae5f7ba94/YLv44PZM6tw1sxACY1-U_.png
Changyao Tian
user
Changyao
687dd61e2e8db0930be6f10d
Hao Li
687dd61e2e8db0930be6f10e
Weiyun Wang
687dd61e2e8db0930be6f10f
Wenhai Wang
687dd61e2e8db0930be6f110
Xizhou Zhu
admin_assigned
2025-07-21T15:58:49.646Z
64ae2359179421d320b1694b
/avatars/c387a75191005bcaa473091de5383a10.svg
Xizhou Zhu
user
Einsiedler
687dd61e2e8db0930be6f111
Yu Qiao
687dd61e2e8db0930be6f112
Jifeng Dai
admin_assigned
2025-07-21T15:58:43.488Z
64686f7172d9180d4ac8b4e4
/avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
Jifeng Dai
user
daijifeng
Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal
  Large Language Models
This paper focuses on monolithic Multimodal Large Language Models (MLLMs),
which integrate visual encoding and language decoding into a single model.
Existing structures and pre-training strategies for monolithic MLLMs often
suffer from unstable optimization and catastrophic forgetting. To address these
challenges, our key idea is to embed a new visual parameter space into a
pre-trained LLM, enabling stable learning of visual knowledge from noisy data
via delta tuning. Based on this principle, we first introduce Mono-InternVL, an
advanced monolithic MLLM that incorporates a set of visual experts through a
multimodal mixture-of-experts architecture. In addition, we design an
innovative Endogenous Visual Pre-training (EViP) for Mono-InternVL to maximize
its visual capabilities via progressive learning. Mono-InternVL achieves
competitive performance against existing MLLMs but also leads to relatively
expensive data cost. Therefore, we further present Mono-InternVL-1.5, a cheaper
and stronger monolithic MLLM equipped with an improved EViP (EViP++). EViP++
introduces additional visual attention experts to Mono-InternVL-1.5 and
re-organizes the pre-training process in an efficient manner. During inference,
it includes a fused CUDA kernel to speed up its MoE operations. With these
designs, Mono-InternVL-1.5 significantly reduces training and inference costs,
while still maintaining competitive performance with Mono-InternVL. To evaluate
our approach, we conduct extensive experiments across 15 benchmarks. Results
demonstrate that Mono-InternVL outperforms existing monolithic MLLMs on 12 out
of 15 benchmarks, e.g., +114-point improvement over Emu3 on OCRBench. Compared
to its modular counterpart, i.e., InternVL-1.5, Mono-InternVL-1.5 achieves
similar multimodal performance while reducing first-token latency by up to 69%.
Code and models are released at https://github.com/OpenGVLab/Mono-InternVL.
Mono-InternVL, an advanced monolithic Multimodal Large Language Model, integrates visual experts and employs Endogenous Visual Pre-training to enhance visual learning and reduce computational costs.
Multimodal Large Language Models monolithic MLLMs visual parameter space delta tuning multimodal mixture-of-experts architecture Endogenous Visual Pre-training EViP EViP++ visual attention experts CUDA kernel MoE operations OCRBench first-token latency