title: Mini-Gemini: Mining the Potential of Multi-modality Vision Language
    Models
summary: In this work, we introduce Mini-Gemini, a simple and effective framework
  enhancing multi-modality Vision Language Models (VLMs). Despite the
  advancements in VLMs facilitating basic visual dialog and reasoning, a
  performance gap persists compared to advanced models like GPT-4 and Gemini. We
  try to narrow the gap by mining the potential of VLMs for better performance
  and any-to-any workflow from three aspects, i.e., high-resolution visual
  tokens, high-quality data, and VLM-guided generation. To enhance visual tokens,
  we propose to utilize an additional visual encoder for high-resolution
  refinement without increasing the visual token count. We further construct a
  high-quality dataset that promotes precise image comprehension and
  reasoning-based generation, expanding the operational scope of current VLMs. In
  general, Mini-Gemini further mines the potential of VLMs and empowers current
  frameworks with image understanding, reasoning, and generation simultaneously.
  Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs)
  from 2B to 34B. It is demonstrated to achieve leading performance in several
  zero-shot benchmarks and even surpasses the developed private models. Code and
  models are available at https://github.com/dvlab-research/MiniGemini.
paper_authors._id: 6604d0927e71d1d4b928fa8f
paper_authors.name: Yanwei Li
paper_authors._id: 6604d0927e71d1d4b928fa90
paper_authors.name: Yuechen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T19:35:39.530Z
paper_authors._id: 6418554a0956be7233a1023e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
paper_authors.fullname: zhang yuechen
paper_authors.type: user
paper_authors.user: julianjuaner
paper_authors._id: 6604d0927e71d1d4b928fa91
paper_authors.name: Chengyao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-28T16:39:47.500Z
paper_authors._id: 6423e35b30b0e4ab36dd1b16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
paper_authors.fullname: Wang Chengyao
paper_authors.type: user
paper_authors.user: wcy1122
paper_authors._id: 6604d0927e71d1d4b928fa92
paper_authors.name: Zhisheng Zhong
paper_authors._id: 6604d0927e71d1d4b928fa93
paper_authors.name: Yixin Chen
paper_authors._id: 6604d0927e71d1d4b928fa94
paper_authors.name: Ruihang Chu
paper_authors._id: 6604d0927e71d1d4b928fa95
paper_authors.name: Shaoteng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:51:16.690Z
paper_authors._id: 63ace528403e3d8deaf23c39
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678936054290-63ace528403e3d8deaf23c39.png
paper_authors.fullname: Shaoteng Liu
paper_authors.type: user
paper_authors.user: Shaldon
paper_authors._id: 6604d0927e71d1d4b928fa96
paper_authors.name: Jiaya Jia
paper_title: Mini-Gemini: Mining the Potential of Multi-modality Vision Language
    Models
paper_summary: In this work, we introduce Mini-Gemini, a simple and effective framework
  enhancing multi-modality Vision Language Models (VLMs). Despite the
  advancements in VLMs facilitating basic visual dialog and reasoning, a
  performance gap persists compared to advanced models like GPT-4 and Gemini. We
  try to narrow the gap by mining the potential of VLMs for better performance
  and any-to-any workflow from three aspects, i.e., high-resolution visual
  tokens, high-quality data, and VLM-guided generation. To enhance visual tokens,
  we propose to utilize an additional visual encoder for high-resolution
  refinement without increasing the visual token count. We further construct a
  high-quality dataset that promotes precise image comprehension and
  reasoning-based generation, expanding the operational scope of current VLMs. In
  general, Mini-Gemini further mines the potential of VLMs and empowers current
  frameworks with image understanding, reasoning, and generation simultaneously.
  Mini-Gemini supports a series of dense and MoE Large Language Models (LLMs)
  from 2B to 34B. It is demonstrated to achieve leading performance in several
  zero-shot benchmarks and even surpasses the developed private models. Code and
  models are available at https://github.com/dvlab-research/MiniGemini.
paper_ai_summary: Mini-Gemini enhances multi-modality Vision Language Models through high-resolution visual tokens, high-quality datasets, and VLM-guided generation, achieving leading performance in zero-shot benchmarks.
paper_ai_keywords: Multi-modality Vision Language Models VLMs high-resolution visual tokens visual encoder high-quality data VLM-guided generation LLMs zero-shot benchmarks