title: TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal
    Foundation Models
summary: Existing benchmarks often highlight the remarkable performance achieved by
  state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal
  context for video understanding. However, how well do the models truly perform
  visual temporal reasoning? Our study of existing benchmarks shows that this
  capability of MFMs is likely overestimated as many questions can be solved by
  using a single, few, or out-of-order frames. To systematically examine current
  visual temporal reasoning tasks, we propose three principles with corresponding
  metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame
  Information Disparity. Following these principles, we introduce TOMATO,
  Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to
  rigorously assess MFMs' temporal reasoning capabilities in video understanding.
  TOMATO comprises 1,484 carefully curated, human-annotated questions spanning
  six tasks (i.e., action count, direction, rotation, shape & trend, velocity &
  frequency, and visual cues), applied to 1,417 videos, including 805
  self-recorded and -generated videos, that encompass human-centric, real-world,
  and simulated scenarios. Our comprehensive evaluation reveals a human-model
  performance gap of 57.3% with the best-performing model. Moreover, our in-depth
  analysis uncovers more fundamental limitations beyond this gap in current MFMs.
  While they can accurately recognize events in isolated frames, they fail to
  interpret these frames as a continuous sequence. We believe TOMATO will serve
  as a crucial testbed for evaluating the next-generation MFMs and as a call to
  the community to develop AI systems capable of comprehending human world
  dynamics through the video modality.
paper_authors._id: 6728399d3c7da6e484d66ee7
paper_authors.name: Ziyao Shangguan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:35:55.473Z
paper_authors._id: 65dea56779f827c045b1df96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jtx4JcM5CNgoVyzxTC93S.jpeg
paper_authors.fullname: Ziyao Shangguan
paper_authors.type: user
paper_authors.user: ziyaosg
paper_authors._id: 6728399d3c7da6e484d66ee8
paper_authors.name: Chuhan Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-04T09:35:57.404Z
paper_authors._id: 65415f1d5168c4f3487a2103
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65415f1d5168c4f3487a2103/qJuzDpOGSDL4E1-L2lGWW.jpeg
paper_authors.fullname: Chuhan Li
paper_authors.type: user
paper_authors.user: ChuhanLi
paper_authors._id: 6728399d3c7da6e484d66ee9
paper_authors.name: Yuxuan Ding
paper_authors._id: 6728399d3c7da6e484d66eea
paper_authors.name: Yanan Zheng
paper_authors._id: 6728399d3c7da6e484d66eeb
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:16:40.048Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 6728399d3c7da6e484d66eec
paper_authors.name: Tesca Fitzgerald
paper_authors._id: 6728399d3c7da6e484d66eed
paper_authors.name: Arman Cohan
paper_title: TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal
    Foundation Models
paper_summary: Existing benchmarks often highlight the remarkable performance achieved by
  state-of-the-art Multimodal Foundation Models (MFMs) in leveraging temporal
  context for video understanding. However, how well do the models truly perform
  visual temporal reasoning? Our study of existing benchmarks shows that this
  capability of MFMs is likely overestimated as many questions can be solved by
  using a single, few, or out-of-order frames. To systematically examine current
  visual temporal reasoning tasks, we propose three principles with corresponding
  metrics: (1) Multi-Frame Gain, (2) Frame Order Sensitivity, and (3) Frame
  Information Disparity. Following these principles, we introduce TOMATO,
  Temporal Reasoning Multimodal Evaluation, a novel benchmark crafted to
  rigorously assess MFMs' temporal reasoning capabilities in video understanding.
  TOMATO comprises 1,484 carefully curated, human-annotated questions spanning
  six tasks (i.e., action count, direction, rotation, shape & trend, velocity &
  frequency, and visual cues), applied to 1,417 videos, including 805
  self-recorded and -generated videos, that encompass human-centric, real-world,
  and simulated scenarios. Our comprehensive evaluation reveals a human-model
  performance gap of 57.3% with the best-performing model. Moreover, our in-depth
  analysis uncovers more fundamental limitations beyond this gap in current MFMs.
  While they can accurately recognize events in isolated frames, they fail to
  interpret these frames as a continuous sequence. We believe TOMATO will serve
  as a crucial testbed for evaluating the next-generation MFMs and as a call to
  the community to develop AI systems capable of comprehending human world
  dynamics through the video modality.
paper_ai_summary: TOMATO, a new benchmark, evaluates the temporal reasoning capabilities of Multimodal Foundation Models in video understanding, revealing significant limitations beyond performance metrics.
paper_ai_keywords: Multimodal Foundation Models MFMs visual temporal reasoning temporal reasoning benchmark Multi-Frame Gain Frame Order Sensitivity Frame Information Disparity TOMATO Temporal Reasoning Multimodal Evaluation video understanding action count direction rotation shape & trend velocity & frequency visual cues human-centric real-world simulated scenarios