title: VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit
    Matching Visual Cues
summary: Visually linking matching cues is a crucial ability in daily life, such as
  identifying the same person in multiple photos based on their cues, even
  without knowing who they are. Despite the extensive knowledge that
  vision-language models (VLMs) possess, it remains largely unexplored whether
  they are capable of performing this fundamental task. To address this, we
  introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can
  Visually Link Matching cues, with 9 subtasks and over 3,000 test cases.
  Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with
  further analysis of various language-side and vision-side prompting methods,
  leads to a total of eight key findings. We identify critical challenges in
  models' ability to link visual cues, highlighting a significant performance gap
  where even GPT-4o lags 34.80% behind humans. Based on these insights, we
  advocate for (i) enhancing core visual capabilities to improve adaptability and
  reduce reliance on prior knowledge, (ii) establishing clearer principles for
  integrating language-based reasoning in vision-centric tasks to prevent
  unnecessary biases, and (iii) shifting vision-text training paradigms toward
  fostering models' ability to independently structure and infer relationships
  among visual cues.
paper_authors._id: 67b8922ef6632327952ec1e1
paper_authors.name: Jianshu Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-21T14:48:16.643Z
paper_authors._id: 65d8b0f0661492b25c6623de
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d8b0f0661492b25c6623de/c6LPDse8NIV_3BHIu8dYe.png
paper_authors.fullname: Jianshu Zhang
paper_authors.type: user
paper_authors.user: Sterzhang
paper_authors._id: 67b8922ef6632327952ec1e2
paper_authors.name: Dongyu Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-24T09:20:43.528Z
paper_authors._id: 64b0377121a001042bc0d274
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b0377121a001042bc0d274/Hk8yI5_s7ey5o9SVZzXrB.png
paper_authors.fullname: Dongyu Yao
paper_authors.type: user
paper_authors.user: RainJamesY
paper_authors._id: 67b8922ef6632327952ec1e3
paper_authors.name: Renjie Pi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-24T13:18:30.254Z
paper_authors._id: 63f45b8d520c14618930d175
paper_authors.avatarUrl: /avatars/a20994594579b52a8be8bd2c4acbb913.svg
paper_authors.fullname: renjie
paper_authors.type: user
paper_authors.user: renjiepi
paper_authors._id: 67b8922ef6632327952ec1e4
paper_authors.name: Paul Pu Liang
paper_authors._id: 67b8922ef6632327952ec1e5
paper_authors.name: Yi R.
paper_authors._id: 67b8922ef6632327952ec1e6
paper_authors.name: Fung
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-26T01:33:46.195Z
paper_authors._id: 67be6f4f22513d7d52d7ef66
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/jQEVQufb6BOb9cQpb6NIt.png
paper_authors.fullname: San Yi
paper_authors.type: user
paper_authors.user: Sanyia
paper_title: VLM^2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit
    Matching Visual Cues
paper_summary: Visually linking matching cues is a crucial ability in daily life, such as
  identifying the same person in multiple photos based on their cues, even
  without knowing who they are. Despite the extensive knowledge that
  vision-language models (VLMs) possess, it remains largely unexplored whether
  they are capable of performing this fundamental task. To address this, we
  introduce VLM^2-Bench, a benchmark designed to assess whether VLMs can
  Visually Link Matching cues, with 9 subtasks and over 3,000 test cases.
  Comprehensive evaluation across eight open-source VLMs and GPT-4o, along with
  further analysis of various language-side and vision-side prompting methods,
  leads to a total of eight key findings. We identify critical challenges in
  models' ability to link visual cues, highlighting a significant performance gap
  where even GPT-4o lags 34.80% behind humans. Based on these insights, we
  advocate for (i) enhancing core visual capabilities to improve adaptability and
  reduce reliance on prior knowledge, (ii) establishing clearer principles for
  integrating language-based reasoning in vision-centric tasks to prevent
  unnecessary biases, and (iii) shifting vision-text training paradigms toward
  fostering models' ability to independently structure and infer relationships
  among visual cues.
paper_ai_keywords: vision-language models (VLMs) VLM$^2$-Bench visual cues benchmark subtasks test cases GPT-4o prompting methods visual capabilities language-based reasoning vision-centric tasks biases vision-text training paradigms structure and infer relationships