BLINK: Multimodal Large Language Models Can See but Not Perceive
We introduce Blink, a new benchmark for multimodal language models (LLMs)
that focuses on core visual perception abilities not found in other
evaluations. Most of the Blink tasks can be solved by humans "within a blink"
(e.g., relative depth estimation, visual correspondence, forensics detection,
and multi-view reasoning). However, we find these perception-demanding tasks
cast significant challenges for current multimodal LLMs because they resist
mediation through natural language. Blink reformats 14 classic computer vision
tasks into 3,807 multiple-choice questions, paired with single or multiple
images and visual prompting. While humans get 95.70% accuracy on average, Blink
is surprisingly challenging for existing multimodal LLMs: even the
best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only
13.17% and 7.63% higher than random guessing, indicating that such perception
abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also
highlights that specialist CV models could solve these problems much better,
suggesting potential pathways for future improvements. We believe Blink will
stimulate the community to help multimodal LLMs catch up with human-level
visual perception.
6621c4ca2b5271e442552378
Xingyu Fu
claimed_verified
2024-04-19T07:38:54.318Z
6336091b2db86a181ccd6054
/avatars/829f69436225d05d2c2136bc90f640d7.svg
Xingyu Fu
user
Fiaa
6621c4ca2b5271e442552379
Yushi Hu
admin_assigned
2024-04-19T08:10:19.164Z
62b1474bdcbad6848a91a54e
/avatars/d7308899b46232cad4a48a0e876449a8.svg
Yushi Hu
user
yushihu
6621c4ca2b5271e44255237a
Bangzheng Li
6621c4ca2b5271e44255237b
Yu Feng
6621c4ca2b5271e44255237c
Haoyu Wang
6621c4ca2b5271e44255237d
Xudong Lin
6621c4ca2b5271e44255237e
Dan Roth
6621c4ca2b5271e44255237f
Noah A. Smith
6621c4ca2b5271e442552380
Wei-Chiu Ma
admin_assigned
2024-04-19T08:11:53.754Z
624fb0c8841b0a7f1b583e72
/avatars/a695a528817f57a98c946056d6f9b2f9.svg
Wei-Chiu Ma
user
weichium
6621c4ca2b5271e442552381
Ranjay Krishna
BLINK: Multimodal Large Language Models Can See but Not Perceive
We introduce Blink, a new benchmark for multimodal language models (LLMs)
that focuses on core visual perception abilities not found in other
evaluations. Most of the Blink tasks can be solved by humans "within a blink"
(e.g., relative depth estimation, visual correspondence, forensics detection,
and multi-view reasoning). However, we find these perception-demanding tasks
cast significant challenges for current multimodal LLMs because they resist
mediation through natural language. Blink reformats 14 classic computer vision
tasks into 3,807 multiple-choice questions, paired with single or multiple
images and visual prompting. While humans get 95.70% accuracy on average, Blink
is surprisingly challenging for existing multimodal LLMs: even the
best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only
13.17% and 7.63% higher than random guessing, indicating that such perception
abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also
highlights that specialist CV models could solve these problems much better,
suggesting potential pathways for future improvements. We believe Blink will
stimulate the community to help multimodal LLMs catch up with human-level
visual perception.
Blink is a benchmark for evaluating the visual perception abilities of multimodal language models, demonstrating current limitations compared to human performance and specialist computer vision models.
multimodal language models LLMs visual perception relative depth estimation visual correspondence forensics detection multi-view reasoning computer vision tasks multiple-choice questions GPT-4V Gemini accuracy random guessing specialist CV models