Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to
evaluate the perception and reasoning skills of pre-trained multimodal models
(e.g. Flamingo, BEiT-3, or GPT-4). Compared to existing benchmarks that focus
on computational tasks (e.g. classification, detection or tracking), the
Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and
types of reasoning (descriptive, explanatory, predictive, counterfactual)
across video, audio, and text modalities, to provide a comprehensive and
efficient evaluation tool. The benchmark probes pre-trained models for their
transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
For these purposes, the Perception Test introduces 11.6k real-world videos, 23s
average length, designed to show perceptually interesting situations, filmed by
around 100 participants worldwide. The videos are densely annotated with six
types of labels (multiple-choice and grounded video question-answers, object
and point tracks, temporal action and sound segments), enabling both language
and non-language evaluations. The fine-tuning and validation splits of the
benchmark are publicly available (CC-BY license), in addition to a challenge
server with a held-out test split. Human baseline results compared to
state-of-the-art video QA models show a significant gap in performance (91.4%
vs 43.6%), suggesting that there is significant room for improvement in
multimodal video understanding.
  Dataset, baselines code, and challenge server are available at
https://github.com/deepmind/perception_test
646d616f4e59b82995e254f4
Viorica Pătrăucean
claimed_verified
2023-05-24T15:30:43.316Z
640b53d991d9f65c58a85d4a
/avatars/bea43ffca19cb0705caea3b22f652adc.svg
Vio Patr
user
viorik
646d616f4e59b82995e254f5
Lucas Smaira
646d616f4e59b82995e254f6
Ankush Gupta
646d616f4e59b82995e254f7
Adrià Recasens Continente
646d616f4e59b82995e254f8
Larisa Markeeva
646d616f4e59b82995e254f9
Dylan Banarse
admin_assigned
2023-05-24T08:00:54.630Z
633ac65c9f5846dc43ff2d83
/avatars/1af08c0d8afbae7df6b80d1b76ed8918.svg
Dylan Banarse
user
dylski
646d616f4e59b82995e254fa
Skanda Koppula
646d616f4e59b82995e254fb
Joseph Heyward
646d616f4e59b82995e254fc
Mateusz Malinowski
646d616f4e59b82995e254fd
Yi Yang
646d616f4e59b82995e254fe
Carl Doersch
646d616f4e59b82995e254ff
Tatiana Matejovicova
claimed_verified
2025-03-28T14:22:50.691Z
66608363906b039ad6515b14
https://cdn-avatars.huggingface.co/v1/production/uploads/66608363906b039ad6515b14/W-pBJo9eCrLq0tRiu3CC4.jpeg
Tatiana Matejovicova
user
tatianama1
646d616f4e59b82995e25500
Yury Sulsky
646d616f4e59b82995e25501
Antoine Miech
646d616f4e59b82995e25502
Alex Frechette
646d616f4e59b82995e25503
Hanna Klimczak
646d616f4e59b82995e25504
Raphael Koster
646d616f4e59b82995e25505
Junlin Zhang
646d616f4e59b82995e25506
Stephanie Winkler
646d616f4e59b82995e25507
Yusuf Aytar
646d616f4e59b82995e25508
Simon Osindero
646d616f4e59b82995e25509
Dima Damen
646d616f4e59b82995e2550a
Andrew Zisserman
646d616f4e59b82995e2550b
João Carreira
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
We propose a novel multimodal video benchmark - the Perception Test - to
evaluate the perception and reasoning skills of pre-trained multimodal models
(e.g. Flamingo, BEiT-3, or GPT-4). Compared to existing benchmarks that focus
on computational tasks (e.g. classification, detection or tracking), the
Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and
types of reasoning (descriptive, explanatory, predictive, counterfactual)
across video, audio, and text modalities, to provide a comprehensive and
efficient evaluation tool. The benchmark probes pre-trained models for their
transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
For these purposes, the Perception Test introduces 11.6k real-world videos, 23s
average length, designed to show perceptually interesting situations, filmed by
around 100 participants worldwide. The videos are densely annotated with six
types of labels (multiple-choice and grounded video question-answers, object
and point tracks, temporal action and sound segments), enabling both language
and non-language evaluations. The fine-tuning and validation splits of the
benchmark are publicly available (CC-BY license), in addition to a challenge
server with a held-out test split. Human baseline results compared to
state-of-the-art video QA models show a significant gap in performance (91.4%
vs 43.6%), suggesting that there is significant room for improvement in
multimodal video understanding.
  Dataset, baselines code, and challenge server are available at
https://github.com/deepmind/perception_test
The Perception Test evaluates multimodal models' perceptual and reasoning skills across video, audio, and text through real-world videos and diverse annotations.
multimodal models pre-trained models Flamingo BEiT-3 GPT-4 Perception Test perception skills reasoning skills zero-shot few-shot transfer capabilities real-world videos video QA multimodal video understanding