title: Tiny LVLM-eHub: Early Multimodal Experiments with Bard
summary: Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated
  significant progress in tackling complex multimodal tasks. Among these
  cutting-edge developments, Google's Bard stands out for its remarkable
  multimodal capabilities, promoting comprehensive comprehension and reasoning
  across various domains. This work presents an early and holistic evaluation of
  LVLMs' multimodal abilities, with a particular focus on Bard, by proposing a
  lightweight variant of LVLM-eHub, named Tiny LVLM-eHub. In comparison to the
  vanilla version, Tiny LVLM-eHub possesses several appealing properties.
  Firstly, it provides a systematic assessment of six categories of multimodal
  capabilities, including visual perception, visual knowledge acquisition, visual
  reasoning, visual commonsense, object hallucination, and embodied intelligence,
  through quantitative evaluation of 42 standard text-related visual
  benchmarks. Secondly, it conducts an in-depth analysis of LVLMs' predictions
  using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and
  accurate evaluation and exhibits improved alignment with human evaluation
  compared to the word matching approach. Thirdly, it comprises a mere 2.1K
  image-text pairs, facilitating ease of use for practitioners to evaluate their
  own offline LVLMs. Through extensive experimental analysis, this study
  demonstrates that Bard outperforms previous LVLMs in most multimodal
  capabilities except object hallucination, to which Bard is still susceptible.
  Tiny LVLM-eHub serves as a baseline evaluation for various LVLMs and encourages
  innovative strategies aimed at advancing multimodal techniques. Our project is
  publicly available at https://github.com/OpenGVLab/Multi-Modality-Arena.
paper_authors._id: 64d1aaa16f107411dae1ca33
paper_authors.name: Wenqi Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:22:18.268Z
paper_authors._id: 64b3fd42eec33e27dcc4c941
paper_authors.avatarUrl: /avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
paper_authors.fullname: Wenqi Shao
paper_authors.type: user
paper_authors.user: wqshao126
paper_authors._id: 64d1aaa16f107411dae1ca34
paper_authors.name: Yutao Hu
paper_authors._id: 64d1aaa16f107411dae1ca35
paper_authors.name: Peng Gao
paper_authors._id: 64d1aaa16f107411dae1ca36
paper_authors.name: Meng Lei
paper_authors._id: 64d1aaa16f107411dae1ca37
paper_authors.name: Kaipeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-17T07:34:58.234Z
paper_authors._id: 65f1713552c38a91e0a445e8
paper_authors.avatarUrl: /avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
paper_authors.fullname: kaipeng
paper_authors.type: user
paper_authors.user: kpzhang996
paper_authors._id: 64d1aaa16f107411dae1ca38
paper_authors.name: Fanqing Meng
paper_authors._id: 64d1aaa16f107411dae1ca39
paper_authors.name: Peng Xu
paper_authors._id: 64d1aaa16f107411dae1ca3a
paper_authors.name: Siyuan Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-16T07:03:43.119Z
paper_authors._id: 634e4120038b5879133552f5
paper_authors.avatarUrl: /avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
paper_authors.fullname: Siyuan
paper_authors.type: user
paper_authors.user: SiyuanH
paper_authors._id: 64d1aaa16f107411dae1ca3b
paper_authors.name: Hongsheng Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-05T10:50:37.698Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 64d1aaa16f107411dae1ca3c
paper_authors.name: Yu Qiao
paper_authors._id: 64d1aaa16f107411dae1ca3d
paper_authors.name: Ping Luo
paper_title: Tiny LVLM-eHub: Early Multimodal Experiments with Bard
paper_summary: Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated
  significant progress in tackling complex multimodal tasks. Among these
  cutting-edge developments, Google's Bard stands out for its remarkable
  multimodal capabilities, promoting comprehensive comprehension and reasoning
  across various domains. This work presents an early and holistic evaluation of
  LVLMs' multimodal abilities, with a particular focus on Bard, by proposing a
  lightweight variant of LVLM-eHub, named Tiny LVLM-eHub. In comparison to the
  vanilla version, Tiny LVLM-eHub possesses several appealing properties.
  Firstly, it provides a systematic assessment of six categories of multimodal
  capabilities, including visual perception, visual knowledge acquisition, visual
  reasoning, visual commonsense, object hallucination, and embodied intelligence,
  through quantitative evaluation of 42 standard text-related visual
  benchmarks. Secondly, it conducts an in-depth analysis of LVLMs' predictions
  using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and
  accurate evaluation and exhibits improved alignment with human evaluation
  compared to the word matching approach. Thirdly, it comprises a mere 2.1K
  image-text pairs, facilitating ease of use for practitioners to evaluate their
  own offline LVLMs. Through extensive experimental analysis, this study
  demonstrates that Bard outperforms previous LVLMs in most multimodal
  capabilities except object hallucination, to which Bard is still susceptible.
  Tiny LVLM-eHub serves as a baseline evaluation for various LVLMs and encourages
  innovative strategies aimed at advancing multimodal techniques. Our project is
  publicly available at https://github.com/OpenGVLab/Multi-Modality-Arena.
paper_ai_keywords: Large Vision-Language Models (LVLMs) multimodal tasks Bard multimodal capabilities visual perception visual knowledge acquisition visual reasoning visual commonsense object hallucination embodied intelligence text-related visual benchmarks ChatGPT Ensemble Evaluation (CEE) multimodal techniques offline LVLMs