TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal
  Foundation Models
Existing benchmarks often highlight the remarkable performance achieved by
state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal
context for video understanding. However, how well do the models truly perform
visual temporal reasoning? Our study of existing benchmarks shows that this
capability of MFMs is likely overestimated as many questions can be solved by
using a single, few, or out-of-order frames. To systematically examine current
visual temporal reasoning tasks, we propose three principles with corresponding
metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame
Information Disparity. Following these principles, we introduce TOMATO,
Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to
rigorously assess MFMs' temporal reasoning capabilities in video understanding.
TOMATO comprises 1,484 carefully curated, human-annotated questions spanning
six tasks (i.e., action count, direction, rotation, shape & trend, velocity &
frequency, and visual cues), applied to 1,417 videos, including 805
self-recorded and -generated videos, that encompass human-centric, real-world,
and simulated scenarios. Our comprehensive evaluation reveals a human-model
performance gap of 57.3% with the best-performing model. Moreover, our in-depth
analysis uncovers more fundamental limitations beyond this gap in current MFMs.
While they can accurately recognize events in isolated frames, they fail to
interpret these frames as a continuous sequence. We believe TOMATO will serve
as a crucial testbed for evaluating the next-generation MFMs and as a call to
the community to develop AI systems capable of comprehending human world
dynamics through the video modality.
6728399d3c7da6e484d66ee7
Ziyao Shangguan
claimed_verified
2024-11-04T09:35:55.473Z
65dea56779f827c045b1df96
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jtx4JcM5CNgoVyzxTC93S.jpeg
Ziyao Shangguan
user
ziyaosg
6728399d3c7da6e484d66ee8
Chuhan Li
claimed_verified
2024-11-04T09:35:57.404Z
65415f1d5168c4f3487a2103
https://cdn-avatars.huggingface.co/v1/production/uploads/65415f1d5168c4f3487a2103/qJuzDpOGSDL4E1-L2lGWW.jpeg
Chuhan Li
user
ChuhanLi
6728399d3c7da6e484d66ee9
Yuxuan Ding
6728399d3c7da6e484d66eea
Yanan Zheng
6728399d3c7da6e484d66eeb
Yilun Zhao
claimed_verified
2024-12-03T11:16:40.048Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
user
yilunzhao
6728399d3c7da6e484d66eec
Tesca Fitzgerald
6728399d3c7da6e484d66eed
Arman Cohan
TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal
  Foundation Models
Existing benchmarks often highlight the remarkable performance achieved by
state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal
context for video understanding. However, how well do the models truly perform
visual temporal reasoning? Our study of existing benchmarks shows that this
capability of MFMs is likely overestimated as many questions can be solved by
using a single, few, or out-of-order frames. To systematically examine current
visual temporal reasoning tasks, we propose three principles with corresponding
metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame
Information Disparity. Following these principles, we introduce TOMATO,
Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to
rigorously assess MFMs' temporal reasoning capabilities in video understanding.
TOMATO comprises 1,484 carefully curated, human-annotated questions spanning
six tasks (i.e., action count, direction, rotation, shape & trend, velocity &
frequency, and visual cues), applied to 1,417 videos, including 805
self-recorded and -generated videos, that encompass human-centric, real-world,
and simulated scenarios. Our comprehensive evaluation reveals a human-model
performance gap of 57.3% with the best-performing model. Moreover, our in-depth
analysis uncovers more fundamental limitations beyond this gap in current MFMs.
While they can accurately recognize events in isolated frames, they fail to
interpret these frames as a continuous sequence. We believe TOMATO will serve
as a crucial testbed for evaluating the next-generation MFMs and as a call to
the community to develop AI systems capable of comprehending human world
dynamics through the video modality.
TOMATO, a new benchmark, evaluates the temporal reasoning capabilities of Multimodal Foundation Models in video understanding, revealing significant limitations beyond performance metrics.
Multimodal Foundation Models MFMs visual temporal reasoning temporal reasoning benchmark Multi-Frame Gain Frame Order Sensitivity Frame Information Disparity TOMATO Temporal Reasoning Multimodal Evaluation video understanding action count direction rotation shape & trend velocity & frequency visual cues human-centric real-world simulated scenarios