Tiny LVLM-eHub: Early Multimodal Experiments with Bard
Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated
significant progress in tackling complex multimodal tasks. Among these
cutting-edge developments, Google's Bard stands out for its remarkable
multimodal capabilities, promoting comprehensive comprehension and reasoning
across various domains. This work presents an early and holistic evaluation of
LVLMs' multimodal abilities, with a particular focus on Bard, by proposing a
lightweight variant of LVLM-eHub, named Tiny LVLM-eHub. In comparison to the
vanilla version, Tiny LVLM-eHub possesses several appealing properties.
Firstly, it provides a systematic assessment of six categories of multimodal
capabilities, including visual perception, visual knowledge acquisition, visual
reasoning, visual commonsense, object hallucination, and embodied intelligence,
through quantitative evaluation of 42 standard text-related visual
benchmarks. Secondly, it conducts an in-depth analysis of LVLMs' predictions
using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and
accurate evaluation and exhibits improved alignment with human evaluation
compared to the word matching approach. Thirdly, it comprises a mere 2.1K
image-text pairs, facilitating ease of use for practitioners to evaluate their
own offline LVLMs. Through extensive experimental analysis, this study
demonstrates that Bard outperforms previous LVLMs in most multimodal
capabilities except object hallucination, to which Bard is still susceptible.
Tiny LVLM-eHub serves as a baseline evaluation for various LVLMs and encourages
innovative strategies aimed at advancing multimodal techniques. Our project is
publicly available at https://github.com/OpenGVLab/Multi-Modality-Arena.
64d1aaa16f107411dae1ca33
Wenqi Shao
admin_assigned
2023-08-08T15:22:18.268Z
64b3fd42eec33e27dcc4c941
/avatars/5aa1a99468fa61d4b8b0e80b592c4e55.svg
Wenqi Shao
user
wqshao126
64d1aaa16f107411dae1ca34
Yutao Hu
64d1aaa16f107411dae1ca35
Peng Gao
64d1aaa16f107411dae1ca36
Meng Lei
64d1aaa16f107411dae1ca37
Kaipeng Zhang
claimed_verified
2024-06-17T07:34:58.234Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
64d1aaa16f107411dae1ca38
Fanqing Meng
64d1aaa16f107411dae1ca39
Peng Xu
64d1aaa16f107411dae1ca3a
Siyuan Huang
claimed_verified
2024-09-16T07:03:43.119Z
634e4120038b5879133552f5
/avatars/34ec861b4bbf1aecf927a7d6e726c7a4.svg
Siyuan
user
SiyuanH
64d1aaa16f107411dae1ca3b
Hongsheng Li
claimed_verified
2024-02-05T10:50:37.698Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
64d1aaa16f107411dae1ca3c
Yu Qiao
64d1aaa16f107411dae1ca3d
Ping Luo
Tiny LVLM-eHub: Early Multimodal Experiments with Bard
Recent advancements in Large Vision-Language Models (LVLMs) have demonstrated
significant progress in tackling complex multimodal tasks. Among these
cutting-edge developments, Google's Bard stands out for its remarkable
multimodal capabilities, promoting comprehensive comprehension and reasoning
across various domains. This work presents an early and holistic evaluation of
LVLMs' multimodal abilities, with a particular focus on Bard, by proposing a
lightweight variant of LVLM-eHub, named Tiny LVLM-eHub. In comparison to the
vanilla version, Tiny LVLM-eHub possesses several appealing properties.
Firstly, it provides a systematic assessment of six categories of multimodal
capabilities, including visual perception, visual knowledge acquisition, visual
reasoning, visual commonsense, object hallucination, and embodied intelligence,
through quantitative evaluation of 42 standard text-related visual
benchmarks. Secondly, it conducts an in-depth analysis of LVLMs' predictions
using the ChatGPT Ensemble Evaluation (CEE), which leads to a robust and
accurate evaluation and exhibits improved alignment with human evaluation
compared to the word matching approach. Thirdly, it comprises a mere 2.1K
image-text pairs, facilitating ease of use for practitioners to evaluate their
own offline LVLMs. Through extensive experimental analysis, this study
demonstrates that Bard outperforms previous LVLMs in most multimodal
capabilities except object hallucination, to which Bard is still susceptible.
Tiny LVLM-eHub serves as a baseline evaluation for various LVLMs and encourages
innovative strategies aimed at advancing multimodal techniques. Our project is
publicly available at https://github.com/OpenGVLab/Multi-Modality-Arena.
Tiny LVLM-eHub evaluates multimodal abilities of LVLMs using ChatGPT Ensemble Evaluation, demonstrating Bard's strengths across various tasks.
Large Vision-Language Models (LVLMs) Bard multimodal capabilities LVLM-eHub text-related visual benchmarks ChatGPT Ensemble Evaluation (CEE) visual perception visual knowledge acquisition visual reasoning visual commonsense object hallucination embodied intelligence image-text pairs