title: A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
summary: The surge of interest towards Multi-modal Large Language Models (MLLMs),
  e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
  academia and industry. They endow Large Language Models (LLMs) with powerful
  capabilities in visual understanding, enabling them to tackle diverse
  multi-modal tasks. Very recently, Google released Gemini, its newest and most
  capable MLLM built from the ground up for multi-modality. In light of the
  superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
  in multi-modal learning? In this paper, we present a preliminary exploration of
  Gemini Pro's visual understanding proficiency, which comprehensively covers
  four domains: fundamental perception, advanced cognition, challenging vision
  tasks, and various expert capacities. We compare Gemini Pro with the
  state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
  open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
  black-box systems. The qualitative samples indicate that, while GPT-4V and
  Gemini showcase different answering styles and preferences, they can exhibit
  comparable visual reasoning capabilities, and Sphinx still trails behind them
  concerning domain generalizability. Specifically, GPT-4V tends to elaborate
  detailed explanations and intermediate steps, and Gemini prefers to output a
  direct and concise answer. The quantitative evaluation on the popular MME
  benchmark also demonstrates the potential of Gemini to be a strong challenger
  to GPT-4V. Our early investigation of Gemini also observes some common issues
  of MLLMs, indicating that there still remains a considerable distance towards
  artificial general intelligence. Our project for tracking the progress of MLLM
  is released at
  https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
paper_authors._id: 6582505d1c4454dde6d3a6aa
paper_authors.name: Chaoyou Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:12:04.184Z
paper_authors._id: 653b268cd1041ca9188954da
paper_authors.avatarUrl: /avatars/2529a821580f5d87ee46867c5dc5b1e5.svg
paper_authors.fullname: BradyFU
paper_authors.type: user
paper_authors.user: BradyFU
paper_authors._id: 6582505d1c4454dde6d3a6ab
paper_authors.name: Renrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-22T08:40:05.948Z
paper_authors._id: 645b8b2687c79b6ec0bb3b7a
paper_authors.avatarUrl: /avatars/00a9db32a42dc950112bf2593bb109cb.svg
paper_authors.fullname: Renrui
paper_authors.type: user
paper_authors.user: ZrrSkywalker
paper_authors._id: 6582505d1c4454dde6d3a6ac
paper_authors.name: Haojia Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:13:21.749Z
paper_authors._id: 64ffd436d522560505a94b8e
paper_authors.avatarUrl: /avatars/02d4faac40ac203cb5d635cfcb39780c.svg
paper_authors.fullname: Haojia Lin
paper_authors.type: user
paper_authors.user: linhaojia13
paper_authors._id: 6582505d1c4454dde6d3a6ad
paper_authors.name: Zihan Wang
paper_authors._id: 6582505d1c4454dde6d3a6ae
paper_authors.name: Timin Gao
paper_authors._id: 6582505d1c4454dde6d3a6af
paper_authors.name: Yongdong Luo
paper_authors._id: 6582505d1c4454dde6d3a6b0
paper_authors.name: Yubo Huang
paper_authors._id: 6582505d1c4454dde6d3a6b1
paper_authors.name: Zhengye Zhang
paper_authors._id: 6582505d1c4454dde6d3a6b2
paper_authors.name: Longtian Qiu
paper_authors._id: 6582505d1c4454dde6d3a6b3
paper_authors.name: Gaoxiang Ye
paper_authors._id: 6582505d1c4454dde6d3a6b4
paper_authors.name: Yunhang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:19:11.278Z
paper_authors._id: 6483143902f98c3f05aff915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
paper_authors.fullname: 沈云航 Yunhang Shen
paper_authors.type: user
paper_authors.user: shenyunhang
paper_authors._id: 6582505d1c4454dde6d3a6b5
paper_authors.name: Mengdan Zhang
paper_authors._id: 6582505d1c4454dde6d3a6b6
paper_authors.name: Peixian Chen
paper_authors._id: 6582505d1c4454dde6d3a6b7
paper_authors.name: Sirui Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:18:50.406Z
paper_authors._id: 653a55ddae6a4080ce3eb860
paper_authors.avatarUrl: /avatars/44a62d1e0b72a42690d1658e5b42f3a4.svg
paper_authors.fullname: Sirui Zhao
paper_authors.type: user
paper_authors.user: SiruiZhao
paper_authors._id: 6582505d1c4454dde6d3a6b8
paper_authors.name: Xiawu Zheng
paper_authors._id: 6582505d1c4454dde6d3a6b9
paper_authors.name: Shaohui Lin
paper_authors._id: 6582505d1c4454dde6d3a6ba
paper_authors.name: Deqiang Jiang
paper_authors._id: 6582505d1c4454dde6d3a6bb
paper_authors.name: Di Yin
paper_authors._id: 6582505d1c4454dde6d3a6bc
paper_authors.name: Peng Gao
paper_authors._id: 6582505d1c4454dde6d3a6bd
paper_authors.name: Ke Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T20:02:12.226Z
paper_authors._id: 63280915eeee4dd858083092
paper_authors.avatarUrl: /avatars/78347af4af42527d53e88d9969c5c934.svg
paper_authors.fullname: Ke Li
paper_authors.type: user
paper_authors.user: tristanli
paper_authors._id: 6582505d1c4454dde6d3a6be
paper_authors.name: Xing Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-20T09:16:45.196Z
paper_authors._id: 647401e50da364bd0d002f2a
paper_authors.avatarUrl: /avatars/f1586f610f21ddb4f868856208c2cfab.svg
paper_authors.fullname: XING Sun
paper_authors.type: user
paper_authors.user: tedsun
paper_authors._id: 6582505d1c4454dde6d3a6bf
paper_authors.name: Rongrong Ji
paper_title: A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise
paper_summary: The surge of interest towards Multi-modal Large Language Models (MLLMs),
  e.g., GPT-4V(ision) from OpenAI, has marked a significant trend in both
  academia and industry. They endow Large Language Models (LLMs) with powerful
  capabilities in visual understanding, enabling them to tackle diverse
  multi-modal tasks. Very recently, Google released Gemini, its newest and most
  capable MLLM built from the ground up for multi-modality. In light of the
  superior reasoning capabilities, can Gemini challenge GPT-4V's leading position
  in multi-modal learning? In this paper, we present a preliminary exploration of
  Gemini Pro's visual understanding proficiency, which comprehensively covers
  four domains: fundamental perception, advanced cognition, challenging vision
  tasks, and various expert capacities. We compare Gemini Pro with the
  state-of-the-art GPT-4V to evaluate its upper limits, along with the latest
  open-sourced MLLM, Sphinx, which reveals the gap between manual efforts and
  black-box systems. The qualitative samples indicate that, while GPT-4V and
  Gemini showcase different answering styles and preferences, they can exhibit
  comparable visual reasoning capabilities, and Sphinx still trails behind them
  concerning domain generalizability. Specifically, GPT-4V tends to elaborate
  detailed explanations and intermediate steps, and Gemini prefers to output a
  direct and concise answer. The quantitative evaluation on the popular MME
  benchmark also demonstrates the potential of Gemini to be a strong challenger
  to GPT-4V. Our early investigation of Gemini also observes some common issues
  of MLLMs, indicating that there still remains a considerable distance towards
  artificial general intelligence. Our project for tracking the progress of MLLM
  is released at
  https://github.com/BradyFU/Awesome-Multimodal-Large-Language-Models.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) GPT-4V(ision) Gemini Large Language Models (LLMs) Visual understanding multi-modal tasks fundamental perception advanced cognition challenging vision tasks expert capacities open-sourced MLLM Sphinx visual reasoning capabilities MME benchmark artificial general intelligence