Mini-Gemini: Mining the Potential of Multi-modality Vision Language
  Models
In this work, we introduce Mini-Gemini, a simple and effective framework
enhancing multi-modality Vision Language Models (VLMs). Despite the
advancements in VLMs facilitating basic visual dialog and reasoning, a
performance gap persists compared to advanced models like GPT-4 and Gemini. We
try to narrow the gap by mining the potential of VLMs for better performance
and any-to-any workflow from three aspects, i.e., high-resolution visual
tokens, high-quality data, and VLM-guided generation. To enhance visual tokens,
we propose to utilize an additional visual encoder for high-resolution
refinement without increasing the visual token count. We further construct a
high-quality dataset that promotes precise image comprehension and
reasoning-based generation, expanding the operational scope of current VLMs. In
general, Mini-Gemini further mines the potential of VLMs and empowers current
frameworks with image understanding, reasoning, and generation simultaneously.
Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs)
from 2B to 34B. It is demonstrated to achieve leading performance in several
zero-shot benchmarks and even surpasses the developed private models. Code and
models are available at https://github.com/dvlab-research/MiniGemini.
6604d0927e71d1d4b928fa8f
Yanwei Li
6604d0927e71d1d4b928fa90
Yuechen Zhang
claimed_verified
2024-04-02T19:35:39.530Z
6418554a0956be7233a1023e
https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
zhang yuechen
user
julianjuaner
6604d0927e71d1d4b928fa91
Chengyao Wang
claimed_verified
2024-03-28T16:39:47.500Z
6423e35b30b0e4ab36dd1b16
https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
Wang Chengyao
user
wcy1122
6604d0927e71d1d4b928fa92
Zhisheng Zhong
6604d0927e71d1d4b928fa93
Yixin Chen
6604d0927e71d1d4b928fa94
Ruihang Chu
claimed_verified
2025-07-18T07:51:19.882Z
642e3bcb958faf258a40e89c
/avatars/213501def37dc53032cee17e37fcc4c1.svg
Ruihang Chu
user
Ruihang
6604d0927e71d1d4b928fa95
Shaoteng Liu
claimed_verified
2025-01-15T08:51:16.690Z
63ace528403e3d8deaf23c39
https://cdn-avatars.huggingface.co/v1/production/uploads/1678936054290-63ace528403e3d8deaf23c39.png
Shaoteng Liu
user
Shaldon
6604d0927e71d1d4b928fa96
Jiaya Jia
Mini-Gemini: Mining the Potential of Multi-modality Vision Language
  Models
In this work, we introduce Mini-Gemini, a simple and effective framework
enhancing multi-modality Vision Language Models (VLMs). Despite the
advancements in VLMs facilitating basic visual dialog and reasoning, a
performance gap persists compared to advanced models like GPT-4 and Gemini. We
try to narrow the gap by mining the potential of VLMs for better performance
and any-to-any workflow from three aspects, i.e., high-resolution visual
tokens, high-quality data, and VLM-guided generation. To enhance visual tokens,
we propose to utilize an additional visual encoder for high-resolution
refinement without increasing the visual token count. We further construct a
high-quality dataset that promotes precise image comprehension and
reasoning-based generation, expanding the operational scope of current VLMs. In
general, Mini-Gemini further mines the potential of VLMs and empowers current
frameworks with image understanding, reasoning, and generation simultaneously.
Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs)
from 2B to 34B. It is demonstrated to achieve leading performance in several
zero-shot benchmarks and even surpasses the developed private models. Code and
models are available at https://github.com/dvlab-research/MiniGemini.
Mini-Gemini enhances multi-modality Vision Language Models through high-resolution visual tokens, high-quality datasets, and VLM-guided generation, achieving leading performance in zero-shot benchmarks.
Multi-modality Vision Language Models VLMs high-resolution visual tokens visual encoder high-quality data VLM-guided generation LLMs zero-shot benchmarks