Can Vision Language Models Infer Human Gaze Direction? A Controlled
  Study
Gaze-referential inference--the ability to infer what others are looking
at--is a critical component of a theory of mind that underpins natural human-AI
interaction. In a controlled study, we evaluated this skill across 111 Vision
Language Models (VLMs) using photos taken with manipulated difficulty and
variability, comparing performance with that of human participants (N = 65),
and analyzed behaviors using mixed-effects models. We found that 94 of the 111
VLMs failed to do better than random guessing, while humans achieved
near-ceiling accuracy. VLMs even respond with each choice almost equally
frequently. Are they randomly guessing? Although most VLMs struggle, when we
zoom in on five of the top-tier VLMs with above-chance performance, we find
that their performance declined with increasing task difficulty but varied only
slightly across different prompts and scene objects. These behavioral features
cannot be explained by considering them as random guessers. Instead, they
likely use a combination of heuristics and guessing such that their performance
is subject to the task difficulty but robust to perceptual variations. This
suggests that VLMs, lacking gaze inference capability, have yet to become
technologies that can naturally interact with humans, but the potential
remains.
684ae322dbd21a9cc27b11e0
Zory Zhang
claimed_verified
2025-06-13T07:40:04.583Z
6595f19f2ae03e58169be1b1
/avatars/97c583dbe497b5974c52cc35bcd63e20.svg
Zory Zhang
user
Zory
684ae322dbd21a9cc27b11e1
Pinyuan Feng
684ae322dbd21a9cc27b11e2
Bingyang Wang
684ae322dbd21a9cc27b11e3
Tianwei Zhao
684ae322dbd21a9cc27b11e4
Suyang Yu
684ae322dbd21a9cc27b11e5
Qingying Gao
684ae322dbd21a9cc27b11e6
Hokin Deng
684ae322dbd21a9cc27b11e7
Ziqiao Ma
claimed_verified
2025-06-25T08:09:42.795Z
630cfc45b66f088d547b2768
https://cdn-avatars.huggingface.co/v1/production/uploads/630cfc45b66f088d547b2768/ge2BZLDYeTlpnlIlraHNw.png
Martin Ziqiao Ma
user
marstin
684ae322dbd21a9cc27b11e8
Yijiang Li
684ae322dbd21a9cc27b11e9
Dezhi Luo
Can Vision Language Models Infer Human Gaze Direction? A Controlled
  Study
Gaze-referential inference--the ability to infer what others are looking
at--is a critical component of a theory of mind that underpins natural human-AI
interaction. In a controlled study, we evaluated this skill across 111 Vision
Language Models (VLMs) using photos taken with manipulated difficulty and
variability, comparing performance with that of human participants (N = 65),
and analyzed behaviors using mixed-effects models. We found that 94 of the 111
VLMs failed to do better than random guessing, while humans achieved
near-ceiling accuracy. VLMs even respond with each choice almost equally
frequently. Are they randomly guessing? Although most VLMs struggle, when we
zoom in on five of the top-tier VLMs with above-chance performance, we find
that their performance declined with increasing task difficulty but varied only
slightly across different prompts and scene objects. These behavioral features
cannot be explained by considering them as random guessers. Instead, they
likely use a combination of heuristics and guessing such that their performance
is subject to the task difficulty but robust to perceptual variations. This
suggests that VLMs, lacking gaze inference capability, have yet to become
technologies that can naturally interact with humans, but the potential
remains.
Vision Language Models struggle with gaze-referential inference compared to humans, showing near-random guessing behavior, while top-tier models perform above chance but are still affected by task difficulty.
Vision Language Models gaze-referential inference mixed-effects models task difficulty