  Multimodal counterfactual reasoning is a vital yet challenging ability for AI
systems. It involves predicting the outcomes of hypothetical circumstances
based on vision and language inputs, which enables AI models to learn from
failures and explore hypothetical scenarios. Despite its importance, there are
only a few datasets targeting the counterfactual reasoning abilities of
multimodal models. Among them, they only cover reasoning over synthetic
environments or specific types of events (e.g. traffic collisions), making them
hard to reliably benchmark the model generalization ability in diverse
real-world scenarios and reasoning dimensions. To overcome these limitations,
we develop a video question answering dataset, ACQUIRED: it consists of 3.9K
annotated videos, encompassing a wide range of event types and incorporating
both first and third-person viewpoints, which ensures a focus on real-world
diversity. In addition, each video is annotated with questions that span three
distinct dimensions of reasoning, including physical, social, and temporal,
which can comprehensively evaluate the model counterfactual abilities along
multiple aspects. We benchmark our dataset against several state-of-the-art
language-only and multimodal models and experimental results demonstrate a
significant performance gap (>13%) between models and humans. The findings
suggest that multimodal counterfactual reasoning remains an open challenge and
ACQUIRED is a comprehensive and reliable benchmark for inspiring future
research in this direction.

Computer Vision and Pattern Recognition;Computation and Language