title: Evaluating Multiview Object Consistency in Humans and Image Models
summary: We introduce a benchmark to directly evaluate the alignment between human
  observers and vision models on a 3D shape inference task. We leverage an
  experimental design from the cognitive sciences which requires zero-shot visual
  inferences about object shape: given a set of images, participants identify
  which contain the same/different objects, despite considerable viewpoint
  variation. We draw from a diverse range of images that include common objects
  (e.g., chairs) as well as abstract shapes (i.e., procedurally generated
  `nonsense' objects). After constructing over 2000 unique image sets, we
  administer these tasks to human participants, collecting 35K trials of
  behavioral data from over 500 participants. This includes explicit choice
  behaviors as well as intermediate measures, such as reaction time and gaze
  data. We then evaluate the performance of common vision models (e.g., DINOv2,
  MAE, CLIP). We find that humans outperform all models by a wide margin. Using a
  multi-scale evaluation approach, we identify underlying similarities and
  differences between models and humans: while human-model performance is
  correlated, humans allocate more time/processing on challenging trials. All
  images, data, and code can be accessed via our project page.
paper_authors._id: 66e00d4abb66849b018e6882
paper_authors.name: Tyler Bonnen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:54:32.168Z
paper_authors._id: 63f3ffee4745321de35792d0
paper_authors.avatarUrl: /avatars/93440782a28c5963e93f74038af50b24.svg
paper_authors.fullname: tyler bonnen
paper_authors.type: user
paper_authors.user: tzler
paper_authors._id: 66e00d4abb66849b018e6883
paper_authors.name: Stephanie Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:54:40.585Z
paper_authors._id: 6484d39cdac774c38f687867
paper_authors.avatarUrl: /avatars/559128c3541c0927e7714dbc1daac253.svg
paper_authors.fullname: Stephanie Fu
paper_authors.type: user
paper_authors.user: stephaniefu
paper_authors._id: 66e00d4abb66849b018e6884
paper_authors.name: Yutong Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:54:49.958Z
paper_authors._id: 6332253749a95639154cc894
paper_authors.avatarUrl: /avatars/f8e53fff5a324591026114c431cd407e.svg
paper_authors.fullname: Yutong Bai
paper_authors.type: user
paper_authors.user: Emma02
paper_authors._id: 66e00d4abb66849b018e6885
paper_authors.name: Thomas O'Connell
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:55:09.081Z
paper_authors._id: 63688bd144e19ccad2169b1b
paper_authors.avatarUrl: /avatars/f488303d1a5a6b49a95c6570712e2c28.svg
paper_authors.fullname: Thomas O'Connell
paper_authors.type: user
paper_authors.user: tgoconnell
paper_authors._id: 66e00d4abb66849b018e6886
paper_authors.name: Yoni Friedman
paper_authors._id: 66e00d4abb66849b018e6887
paper_authors.name: Nancy Kanwisher
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-10T09:55:39.051Z
paper_authors._id: 630e7f2264f1f8d0c7728f9e
paper_authors.avatarUrl: /avatars/71e90959b9a89cafc9a53322ce244c4e.svg
paper_authors.fullname: Nancy Kanwisher
paper_authors.type: user
paper_authors.user: Kanwisher
paper_authors._id: 66e00d4abb66849b018e6888
paper_authors.name: Joshua B. Tenenbaum
paper_authors._id: 66e00d4abb66849b018e6889
paper_authors.name: Alexei A. Efros
paper_title: Evaluating Multiview Object Consistency in Humans and Image Models
paper_summary: We introduce a benchmark to directly evaluate the alignment between human
  observers and vision models on a 3D shape inference task. We leverage an
  experimental design from the cognitive sciences which requires zero-shot visual
  inferences about object shape: given a set of images, participants identify
  which contain the same/different objects, despite considerable viewpoint
  variation. We draw from a diverse range of images that include common objects
  (e.g., chairs) as well as abstract shapes (i.e., procedurally generated
  `nonsense' objects). After constructing over 2000 unique image sets, we
  administer these tasks to human participants, collecting 35K trials of
  behavioral data from over 500 participants. This includes explicit choice
  behaviors as well as intermediate measures, such as reaction time and gaze
  data. We then evaluate the performance of common vision models (e.g., DINOv2,
  MAE, CLIP). We find that humans outperform all models by a wide margin. Using a
  multi-scale evaluation approach, we identify underlying similarities and
  differences between models and humans: while human-model performance is
  correlated, humans allocate more time/processing on challenging trials. All
  images, data, and code can be accessed via our project page.
paper_ai_keywords: zero-shot visual viewpoint variation procedural generated nonsense' objects behavioral data reaction time gaze data DINOv2 MAE CLIP multi-scale evaluation