title: Perception Test: A Diagnostic Benchmark for Multimodal Video Models
summary: We propose a novel multimodal video benchmark - the Perception Test - to
  evaluate the perception and reasoning skills of pre-trained multimodal models
  (e.g. Flamingo, BEiT-3, or GPT-4). Compared to existing benchmarks that focus
  on computational tasks (e.g. classification, detection or tracking), the
  Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and
  types of reasoning (descriptive, explanatory, predictive, counterfactual)
  across video, audio, and text modalities, to provide a comprehensive and
  efficient evaluation tool. The benchmark probes pre-trained models for their
  transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  For these purposes, the Perception Test introduces 11.6k real-world videos, 23s
  average length, designed to show perceptually interesting situations, filmed by
  around 100 participants worldwide. The videos are densely annotated with six
  types of labels (multiple-choice and grounded video question-answers, object
  and point tracks, temporal action and sound segments), enabling both language
  and non-language evaluations. The fine-tuning and validation splits of the
  benchmark are publicly available (CC-BY license), in addition to a challenge
  server with a held-out test split. Human baseline results compared to
  state-of-the-art video QA models show a significant gap in performance (91.4%
  vs 43.6%), suggesting that there is significant room for improvement in
  multimodal video understanding.
    Dataset, baselines code, and challenge server are available at
  https://github.com/deepmind/perception_test
paper_authors._id: 646d616f4e59b82995e254f4
paper_authors.name: Viorica Pătrăucean
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-24T15:30:43.316Z
paper_authors._id: 640b53d991d9f65c58a85d4a
paper_authors.avatarUrl: /avatars/bea43ffca19cb0705caea3b22f652adc.svg
paper_authors.fullname: Vio Patr
paper_authors.type: user
paper_authors.user: viorik
paper_authors._id: 646d616f4e59b82995e254f5
paper_authors.name: Lucas Smaira
paper_authors._id: 646d616f4e59b82995e254f6
paper_authors.name: Ankush Gupta
paper_authors._id: 646d616f4e59b82995e254f7
paper_authors.name: Adrià Recasens Continente
paper_authors._id: 646d616f4e59b82995e254f8
paper_authors.name: Larisa Markeeva
paper_authors._id: 646d616f4e59b82995e254f9
paper_authors.name: Dylan Banarse
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-24T08:00:54.630Z
paper_authors._id: 633ac65c9f5846dc43ff2d83
paper_authors.avatarUrl: /avatars/1af08c0d8afbae7df6b80d1b76ed8918.svg
paper_authors.fullname: Dylan Banarse
paper_authors.type: user
paper_authors.user: dylski
paper_authors._id: 646d616f4e59b82995e254fa
paper_authors.name: Skanda Koppula
paper_authors._id: 646d616f4e59b82995e254fb
paper_authors.name: Joseph Heyward
paper_authors._id: 646d616f4e59b82995e254fc
paper_authors.name: Mateusz Malinowski
paper_authors._id: 646d616f4e59b82995e254fd
paper_authors.name: Yi Yang
paper_authors._id: 646d616f4e59b82995e254fe
paper_authors.name: Carl Doersch
paper_authors._id: 646d616f4e59b82995e254ff
paper_authors.name: Tatiana Matejovicova
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-28T14:22:50.691Z
paper_authors._id: 66608363906b039ad6515b14
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66608363906b039ad6515b14/W-pBJo9eCrLq0tRiu3CC4.jpeg
paper_authors.fullname: Tatiana Matejovicova
paper_authors.type: user
paper_authors.user: tatianama1
paper_authors._id: 646d616f4e59b82995e25500
paper_authors.name: Yury Sulsky
paper_authors._id: 646d616f4e59b82995e25501
paper_authors.name: Antoine Miech
paper_authors._id: 646d616f4e59b82995e25502
paper_authors.name: Alex Frechette
paper_authors._id: 646d616f4e59b82995e25503
paper_authors.name: Hanna Klimczak
paper_authors._id: 646d616f4e59b82995e25504
paper_authors.name: Raphael Koster
paper_authors._id: 646d616f4e59b82995e25505
paper_authors.name: Junlin Zhang
paper_authors._id: 646d616f4e59b82995e25506
paper_authors.name: Stephanie Winkler
paper_authors._id: 646d616f4e59b82995e25507
paper_authors.name: Yusuf Aytar
paper_authors._id: 646d616f4e59b82995e25508
paper_authors.name: Simon Osindero
paper_authors._id: 646d616f4e59b82995e25509
paper_authors.name: Dima Damen
paper_authors._id: 646d616f4e59b82995e2550a
paper_authors.name: Andrew Zisserman
paper_authors._id: 646d616f4e59b82995e2550b
paper_authors.name: João Carreira
paper_title: Perception Test: A Diagnostic Benchmark for Multimodal Video Models
paper_summary: We propose a novel multimodal video benchmark - the Perception Test - to
  evaluate the perception and reasoning skills of pre-trained multimodal models
  (e.g. Flamingo, BEiT-3, or GPT-4). Compared to existing benchmarks that focus
  on computational tasks (e.g. classification, detection or tracking), the
  Perception Test focuses on skills (Memory, Abstraction, Physics, Semantics) and
  types of reasoning (descriptive, explanatory, predictive, counterfactual)
  across video, audio, and text modalities, to provide a comprehensive and
  efficient evaluation tool. The benchmark probes pre-trained models for their
  transfer capabilities, in a zero-shot / few-shot or limited finetuning regime.
  For these purposes, the Perception Test introduces 11.6k real-world videos, 23s
  average length, designed to show perceptually interesting situations, filmed by
  around 100 participants worldwide. The videos are densely annotated with six
  types of labels (multiple-choice and grounded video question-answers, object
  and point tracks, temporal action and sound segments), enabling both language
  and non-language evaluations. The fine-tuning and validation splits of the
  benchmark are publicly available (CC-BY license), in addition to a challenge
  server with a held-out test split. Human baseline results compared to
  state-of-the-art video QA models show a significant gap in performance (91.4%
  vs 43.6%), suggesting that there is significant room for improvement in
  multimodal video understanding.
    Dataset, baselines code, and challenge server are available at
  https://github.com/deepmind/perception_test
paper_ai_summary: The Perception Test evaluates multimodal models' perceptual and reasoning skills across video, audio, and text through real-world videos and diverse annotations.
paper_ai_keywords: multimodal models pre-trained models Flamingo BEiT-3 GPT-4 Perception Test perception skills reasoning skills zero-shot few-shot transfer capabilities real-world videos video QA multimodal video understanding