Evaluating Multiview Object Consistency in Humans and Image Models
We introduce a benchmark to directly evaluate the alignment between human
observers and vision models on a 3D shape inference task. We leverage an
experimental design from the cognitive sciences which requires zero-shot visual
inferences about object shape: given a set of images, participants identify
which contain the same/different objects, despite considerable viewpoint
variation. We draw from a diverse range of images that include common objects
(e.g., chairs) as well as abstract shapes (i.e., procedurally generated
`nonsense' objects). After constructing over 2000 unique image sets, we
administer these tasks to human participants, collecting 35K trials of
behavioral data from over 500 participants. This includes explicit choice
behaviors as well as intermediate measures, such as reaction time and gaze
data. We then evaluate the performance of common vision models (e.g., DINOv2,
MAE, CLIP). We find that humans outperform all models by a wide margin. Using a
multi-scale evaluation approach, we identify underlying similarities and
differences between models and humans: while human-model performance is
correlated, humans allocate more time/processing on challenging trials. All
images, data, and code can be accessed via our project page.
66e00d4abb66849b018e6882
Tyler Bonnen
admin_assigned
2024-09-10T09:54:32.168Z
63f3ffee4745321de35792d0
/avatars/93440782a28c5963e93f74038af50b24.svg
tyler bonnen
user
tzler
66e00d4abb66849b018e6883
Stephanie Fu
admin_assigned
2024-09-10T09:54:40.585Z
6484d39cdac774c38f687867
/avatars/559128c3541c0927e7714dbc1daac253.svg
Stephanie Fu
user
stephaniefu
66e00d4abb66849b018e6884
Yutong Bai
admin_assigned
2024-09-10T09:54:49.958Z
6332253749a95639154cc894
/avatars/f8e53fff5a324591026114c431cd407e.svg
Yutong Bai
user
Emma02
66e00d4abb66849b018e6885
Thomas O'Connell
admin_assigned
2024-09-10T09:55:09.081Z
63688bd144e19ccad2169b1b
/avatars/f488303d1a5a6b49a95c6570712e2c28.svg
Thomas O'Connell
user
tgoconnell
66e00d4abb66849b018e6886
Yoni Friedman
66e00d4abb66849b018e6887
Nancy Kanwisher
admin_assigned
2024-09-10T09:55:39.051Z
630e7f2264f1f8d0c7728f9e
/avatars/71e90959b9a89cafc9a53322ce244c4e.svg
Nancy Kanwisher
user
Kanwisher
66e00d4abb66849b018e6888
Joshua B. Tenenbaum
66e00d4abb66849b018e6889
Alexei A. Efros
Evaluating Multiview Object Consistency in Humans and Image Models
We introduce a benchmark to directly evaluate the alignment between human
observers and vision models on a 3D shape inference task. We leverage an
experimental design from the cognitive sciences which requires zero-shot visual
inferences about object shape: given a set of images, participants identify
which contain the same/different objects, despite considerable viewpoint
variation. We draw from a diverse range of images that include common objects
(e.g., chairs) as well as abstract shapes (i.e., procedurally generated
`nonsense' objects). After constructing over 2000 unique image sets, we
administer these tasks to human participants, collecting 35K trials of
behavioral data from over 500 participants. This includes explicit choice
behaviors as well as intermediate measures, such as reaction time and gaze
data. We then evaluate the performance of common vision models (e.g., DINOv2,
MAE, CLIP). We find that humans outperform all models by a wide margin. Using a
multi-scale evaluation approach, we identify underlying similarities and
differences between models and humans: while human-model performance is
correlated, humans allocate more time/processing on challenging trials. All
images, data, and code can be accessed via our project page.
Human performance significantly outperforms common vision models in a zero-shot 3D shape inference task with varied viewpoints, as evaluated using a cognitive science-based benchmark.
DINOv2 MAE CLIP multi-scale evaluation viewpoint variation zero-shot inference