ZeroBench: An Impossible Visual Benchmark for Contemporary Large
  Multimodal Models
Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting
images and, by some measures, have poorer spatial cognition than small children
or animals. Despite this, they attain high scores on many popular visual
benchmarks, with headroom rapidly eroded by an ongoing surge of model progress.
To address this, there is a pressing need for difficult benchmarks that remain
relevant for longer. We take this idea to its limit by introducing ZeroBench-a
lightweight visual reasoning benchmark that is entirely impossible for
contemporary frontier LMMs. Our benchmark consists of 100 manually curated
questions and 334 less difficult subquestions. We evaluate 20 LMMs on
ZeroBench, all of which score 0.0%, and rigorously analyse the errors. To
encourage progress in visual understanding, we publicly release ZeroBench.
67b2aae22a4cd186392a18b2
Jonathan Roberts
claimed_verified
2025-02-18T09:33:01.809Z
632456d21ed511c0c5231afd
https://cdn-avatars.huggingface.co/v1/production/uploads/1671535517748-632456d21ed511c0c5231afd.jpeg
Jonathan Roberts
user
jonathan-roberts1
67b2aae22a4cd186392a18b3
Mohammad Reza Taesiri
claimed_verified
2025-02-18T09:33:04.484Z
6039478ab3ecf716b1a5fd4d
https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg
taesiri
user
taesiri
67b2aae22a4cd186392a18b4
Ansh Sharma
67b2aae22a4cd186392a18b5
Akash Gupta
67b2aae22a4cd186392a18b6
Samuel Roberts
67b2aae22a4cd186392a18b7
Ioana Croitoru
67b2aae22a4cd186392a18b8
Simion-Vlad Bogolin
67b2aae22a4cd186392a18b9
Jialu Tang
67b2aae22a4cd186392a18ba
Florian Langer
67b2aae22a4cd186392a18bb
Vyas Raina
67b2aae22a4cd186392a18bc
Vatsal Raina
67b2aae22a4cd186392a18bd
Hanyi Xiong
67b2aae22a4cd186392a18be
Vishaal Udandarao
67b2aae22a4cd186392a18bf
Jingyi Lu
67b2aae22a4cd186392a18c0
Shiyang Chen
67b2aae22a4cd186392a18c1
Sam Purkis
67b2aae22a4cd186392a18c2
Tianshuo Yan
67b2aae22a4cd186392a18c3
Wenye Lin
claimed_verified
2025-04-01T07:47:51.285Z
5fdcb57b0768642d6ebe7e7c
https://cdn-avatars.huggingface.co/v1/production/uploads/1608299846124-noauth.jpeg
Linius Lin
user
linius
67b2aae22a4cd186392a18c4
Gyungin Shin
67b2aae22a4cd186392a18c5
Qiaochu Yang
67b2aae22a4cd186392a18c6
Anh Totti Nguyen
claimed_verified
2025-06-23T08:34:05.162Z
60e85b3fcd1cf4e418fff651
https://cdn-avatars.huggingface.co/v1/production/uploads/60e85b3fcd1cf4e418fff651/TUaEA4SqB6xuG2JhwNETC.jpeg
Anh (Totti) Nguyen
user
anhng8
67b2aae22a4cd186392a18c7
Kai Han
67b2aae22a4cd186392a18c8
Samuel Albanie
ZeroBench: An Impossible Visual Benchmark for Contemporary Large
  Multimodal Models
Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting
images and, by some measures, have poorer spatial cognition than small children
or animals. Despite this, they attain high scores on many popular visual
benchmarks, with headroom rapidly eroded by an ongoing surge of model progress.
To address this, there is a pressing need for difficult benchmarks that remain
relevant for longer. We take this idea to its limit by introducing ZeroBench-a
lightweight visual reasoning benchmark that is entirely impossible for
contemporary frontier LMMs. Our benchmark consists of 100 manually curated
questions and 334 less difficult subquestions. We evaluate 20 LMMs on
ZeroBench, all of which score 0.0%, and rigorously analyse the errors. To
encourage progress in visual understanding, we publicly release ZeroBench.
ZeroBench, a new visual reasoning benchmark, demonstrates the limitations of current large multimodal models in understanding images through a set of questions they cannot solve.
Large Multimodal Models (LMMs) visual reasoning benchmark spatial cognition visual benchmarks ZeroBench