title: ZeroBench: An Impossible Visual Benchmark for Contemporary Large
    Multimodal Models
summary: Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting
  images and, by some measures, have poorer spatial cognition than small children
  or animals. Despite this, they attain high scores on many popular visual
  benchmarks, with headroom rapidly eroded by an ongoing surge of model progress.
  To address this, there is a pressing need for difficult benchmarks that remain
  relevant for longer. We take this idea to its limit by introducing ZeroBench-a
  lightweight visual reasoning benchmark that is entirely impossible for
  contemporary frontier LMMs. Our benchmark consists of 100 manually curated
  questions and 334 less difficult subquestions. We evaluate 20 LMMs on
  ZeroBench, all of which score 0.0%, and rigorously analyse the errors. To
  encourage progress in visual understanding, we publicly release ZeroBench.
paper_authors._id: 67b2aae22a4cd186392a18b2
paper_authors.name: Jonathan Roberts
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:33:01.809Z
paper_authors._id: 632456d21ed511c0c5231afd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1671535517748-632456d21ed511c0c5231afd.jpeg
paper_authors.fullname: Jonathan Roberts
paper_authors.type: user
paper_authors.user: jonathan-roberts1
paper_authors._id: 67b2aae22a4cd186392a18b3
paper_authors.name: Mohammad Reza Taesiri
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:33:04.484Z
paper_authors._id: 6039478ab3ecf716b1a5fd4d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6039478ab3ecf716b1a5fd4d/_Thy4E7taiSYBLKxEKJbT.jpeg
paper_authors.fullname: taesiri
paper_authors.type: user
paper_authors.user: taesiri
paper_authors._id: 67b2aae22a4cd186392a18b4
paper_authors.name: Ansh Sharma
paper_authors._id: 67b2aae22a4cd186392a18b5
paper_authors.name: Akash Gupta
paper_authors._id: 67b2aae22a4cd186392a18b6
paper_authors.name: Samuel Roberts
paper_authors._id: 67b2aae22a4cd186392a18b7
paper_authors.name: Ioana Croitoru
paper_authors._id: 67b2aae22a4cd186392a18b8
paper_authors.name: Simion-Vlad Bogolin
paper_authors._id: 67b2aae22a4cd186392a18b9
paper_authors.name: Jialu Tang
paper_authors._id: 67b2aae22a4cd186392a18ba
paper_authors.name: Florian Langer
paper_authors._id: 67b2aae22a4cd186392a18bb
paper_authors.name: Vyas Raina
paper_authors._id: 67b2aae22a4cd186392a18bc
paper_authors.name: Vatsal Raina
paper_authors._id: 67b2aae22a4cd186392a18bd
paper_authors.name: Hanyi Xiong
paper_authors._id: 67b2aae22a4cd186392a18be
paper_authors.name: Vishaal Udandarao
paper_authors._id: 67b2aae22a4cd186392a18bf
paper_authors.name: Jingyi Lu
paper_authors._id: 67b2aae22a4cd186392a18c0
paper_authors.name: Shiyang Chen
paper_authors._id: 67b2aae22a4cd186392a18c1
paper_authors.name: Sam Purkis
paper_authors._id: 67b2aae22a4cd186392a18c2
paper_authors.name: Tianshuo Yan
paper_authors._id: 67b2aae22a4cd186392a18c3
paper_authors.name: Wenye Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-01T07:47:51.285Z
paper_authors._id: 5fdcb57b0768642d6ebe7e7c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1608299846124-noauth.jpeg
paper_authors.fullname: Linius Lin
paper_authors.type: user
paper_authors.user: linius
paper_authors._id: 67b2aae22a4cd186392a18c4
paper_authors.name: Gyungin Shin
paper_authors._id: 67b2aae22a4cd186392a18c5
paper_authors.name: Qiaochu Yang
paper_authors._id: 67b2aae22a4cd186392a18c6
paper_authors.name: Anh Totti Nguyen
paper_authors._id: 67b2aae22a4cd186392a18c7
paper_authors.name: Kai Han
paper_authors._id: 67b2aae22a4cd186392a18c8
paper_authors.name: Samuel Albanie
paper_title: ZeroBench: An Impossible Visual Benchmark for Contemporary Large
    Multimodal Models
paper_summary: Large Multimodal Models (LMMs) exhibit major shortfalls when interpreting
  images and, by some measures, have poorer spatial cognition than small children
  or animals. Despite this, they attain high scores on many popular visual
  benchmarks, with headroom rapidly eroded by an ongoing surge of model progress.
  To address this, there is a pressing need for difficult benchmarks that remain
  relevant for longer. We take this idea to its limit by introducing ZeroBench-a
  lightweight visual reasoning benchmark that is entirely impossible for
  contemporary frontier LMMs. Our benchmark consists of 100 manually curated
  questions and 334 less difficult subquestions. We evaluate 20 LMMs on
  ZeroBench, all of which score 0.0%, and rigorously analyse the errors. To
  encourage progress in visual understanding, we publicly release ZeroBench.
paper_ai_summary: ZeroBench, a new visual reasoning benchmark, demonstrates the limitations of current large multimodal models in understanding images through a set of questions they cannot solve.
paper_ai_keywords: Large Multimodal Models (LMMs) visual reasoning benchmark spatial cognition visual benchmarks ZeroBench