Pixels, Patterns, but No Poetry: To See The World like Humans
Achieving human-like perception and reasoning in Multimodal Large Language
Models (MLLMs) remains a central challenge in artificial intelligence. While
recent research has primarily focused on enhancing reasoning capabilities in
MLLMs, a fundamental question persists: Can Multimodal Large Language Models
truly perceive the world as humans do? This paper shifts focus from reasoning
to perception. Rather than constructing benchmarks specifically for reasoning,
we introduce the Turing Eye Test (TET), a challenging perception-oriented
benchmark comprising four diagnostic tasks that evaluate MLLMs' performance on
synthetic images that humans process intuitively. Our findings reveal that
state-of-the-art MLLMs exhibit catastrophic failures on our perceptual tasks
trivial for humans. Both in-context learning and training on language
backbone-effective for previous benchmarks-fail to improve performance on our
tasks, while fine-tuning the vision tower enables rapid adaptation, suggesting
that our benchmark poses challenges for vision tower generalization rather than
for the knowledge and reasoning capabilities of the language backbone-a key gap
between current MLLMs and human perception. We release a representative subset
of TET tasks in this version, and will introduce more diverse tasks and methods
to enhance visual generalization in future work.
6881be07df7c5aafaf37f0e1
Hongcheng Gao
6881be07df7c5aafaf37f0e2
Zihao Huang
6881be07df7c5aafaf37f0e3
Lin Xu
6881be07df7c5aafaf37f0e4
Jingyi Tang
6881be07df7c5aafaf37f0e5
Xinhao Li
6881be07df7c5aafaf37f0e6
Yue Liu
6881be07df7c5aafaf37f0e7
Haoyang Li
6881be07df7c5aafaf37f0e8
Taihang Hu
6881be07df7c5aafaf37f0e9
Minhua Lin
6881be07df7c5aafaf37f0ea
Xinlong Yang
6881be07df7c5aafaf37f0eb
Ge Wu
6881be07df7c5aafaf37f0ec
Balong Bi
6881be07df7c5aafaf37f0ed
Hongyu Chen
6881be07df7c5aafaf37f0ee
Wentao Zhang
Pixels, Patterns, but No Poetry: To See The World like Humans
Achieving human-like perception and reasoning in Multimodal Large Language
Models (MLLMs) remains a central challenge in artificial intelligence. While
recent research has primarily focused on enhancing reasoning capabilities in
MLLMs, a fundamental question persists: Can Multimodal Large Language Models
truly perceive the world as humans do? This paper shifts focus from reasoning
to perception. Rather than constructing benchmarks specifically for reasoning,
we introduce the Turing Eye Test (TET), a challenging perception-oriented
benchmark comprising four diagnostic tasks that evaluate MLLMs' performance on
synthetic images that humans process intuitively. Our findings reveal that
state-of-the-art MLLMs exhibit catastrophic failures on our perceptual tasks
trivial for humans. Both in-context learning and training on language
backbone-effective for previous benchmarks-fail to improve performance on our
tasks, while fine-tuning the vision tower enables rapid adaptation, suggesting
that our benchmark poses challenges for vision tower generalization rather than
for the knowledge and reasoning capabilities of the language backbone-a key gap
between current MLLMs and human perception. We release a representative subset
of TET tasks in this version, and will introduce more diverse tasks and methods
to enhance visual generalization in future work.
The Turing Eye Test evaluates MLLMs' perceptual capabilities on synthetic images, revealing that vision tower generalization is a key gap compared to human perception.
Multimodal Large Language Models Turing Eye Test in-context learning vision tower visual generalization