title: BLINK: Multimodal Large Language Models Can See but Not Perceive
summary: We introduce Blink, a new benchmark for multimodal language models (LLMs)
  that focuses on core visual perception abilities not found in other
  evaluations. Most of the Blink tasks can be solved by humans "within a blink"
  (e.g., relative depth estimation, visual correspondence, forensics detection,
  and multi-view reasoning). However, we find these perception-demanding tasks
  cast significant challenges for current multimodal LLMs because they resist
  mediation through natural language. Blink reformats 14 classic computer vision
  tasks into 3,807 multiple-choice questions, paired with single or multiple
  images and visual prompting. While humans get 95.70% accuracy on average, Blink
  is surprisingly challenging for existing multimodal LLMs: even the
  best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only
  13.17% and 7.63% higher than random guessing, indicating that such perception
  abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also
  highlights that specialist CV models could solve these problems much better,
  suggesting potential pathways for future improvements. We believe Blink will
  stimulate the community to help multimodal LLMs catch up with human-level
  visual perception.
paper_authors._id: 6621c4ca2b5271e442552378
paper_authors.name: Xingyu Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:38:54.318Z
paper_authors._id: 6336091b2db86a181ccd6054
paper_authors.avatarUrl: /avatars/4d29cef783603dc8ba123f01cde93ad8.svg
paper_authors.fullname: Xingyu Fu
paper_authors.type: user
paper_authors.user: Fiaa
paper_authors._id: 6621c4ca2b5271e442552379
paper_authors.name: Yushi Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:10:19.164Z
paper_authors._id: 62b1474bdcbad6848a91a54e
paper_authors.avatarUrl: /avatars/d7308899b46232cad4a48a0e876449a8.svg
paper_authors.fullname: Yushi Hu
paper_authors.type: user
paper_authors.user: yushihu
paper_authors._id: 6621c4ca2b5271e44255237a
paper_authors.name: Bangzheng Li
paper_authors._id: 6621c4ca2b5271e44255237b
paper_authors.name: Yu Feng
paper_authors._id: 6621c4ca2b5271e44255237c
paper_authors.name: Haoyu Wang
paper_authors._id: 6621c4ca2b5271e44255237d
paper_authors.name: Xudong Lin
paper_authors._id: 6621c4ca2b5271e44255237e
paper_authors.name: Dan Roth
paper_authors._id: 6621c4ca2b5271e44255237f
paper_authors.name: Noah A. Smith
paper_authors._id: 6621c4ca2b5271e442552380
paper_authors.name: Wei-Chiu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-19T08:11:53.754Z
paper_authors._id: 624fb0c8841b0a7f1b583e72
paper_authors.avatarUrl: /avatars/a695a528817f57a98c946056d6f9b2f9.svg
paper_authors.fullname: Wei-Chiu Ma
paper_authors.type: user
paper_authors.user: weichium
paper_authors._id: 6621c4ca2b5271e442552381
paper_authors.name: Ranjay Krishna
paper_title: BLINK: Multimodal Large Language Models Can See but Not Perceive
paper_summary: We introduce Blink, a new benchmark for multimodal language models (LLMs)
  that focuses on core visual perception abilities not found in other
  evaluations. Most of the Blink tasks can be solved by humans "within a blink"
  (e.g., relative depth estimation, visual correspondence, forensics detection,
  and multi-view reasoning). However, we find these perception-demanding tasks
  cast significant challenges for current multimodal LLMs because they resist
  mediation through natural language. Blink reformats 14 classic computer vision
  tasks into 3,807 multiple-choice questions, paired with single or multiple
  images and visual prompting. While humans get 95.70% accuracy on average, Blink
  is surprisingly challenging for existing multimodal LLMs: even the
  best-performing GPT-4V and Gemini achieve accuracies of 51.26% and 45.72%, only
  13.17% and 7.63% higher than random guessing, indicating that such perception
  abilities have not "emerged" yet in recent multimodal LLMs. Our analysis also
  highlights that specialist CV models could solve these problems much better,
  suggesting potential pathways for future improvements. We believe Blink will
  stimulate the community to help multimodal LLMs catch up with human-level
  visual perception.
paper_ai_keywords: multimodal language models (LLMs) visual perception abilities relative depth estimation visual correspondence forensics detection multi-view reasoning multiple-choice questions computer vision tasks visual prompting GPT-4V Gemini