title: Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short
    Videos
summary: There has been growing sentiment recently that modern large multimodal models
  (LMMs) have addressed most of the key challenges related to short video
  comprehension. As a result, both academia and industry are gradually shifting
  their attention towards the more complex challenges posed by understanding
  long-form videos. However, is this really the case? Our studies indicate that
  LMMs still lack many fundamental reasoning capabilities even when dealing with
  short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation
  benchmark encompassing 1000 short and natural video-caption pairs. We
  demonstrate that existing LMMs severely struggle to distinguish temporal
  differences between different actions and object transformations. For example,
  the best model GPT-4o only obtains ~50% on our text and video scores, showing a
  large gap compared to the human baseline of ~90%. All open-source multimodal
  models and CLIP-based models perform much worse, producing mostly random chance
  performance. Through this work, we shed light onto the fact that temporal
  reasoning in short videos is a problem yet to be fully solved. The dataset and
  evaluation code are available at https://vinoground.github.io.
paper_authors._id: 66ff5d6c661a30ae72a1b83b
paper_authors.name: Jianrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T20:11:28.863Z
paper_authors._id: 6508b164abdde5290e5e4939
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6508b164abdde5290e5e4939/lQgAs3BHwCyI7Go1QA62m.jpeg
paper_authors.fullname: Harris Zhang
paper_authors.type: user
paper_authors.user: HanSolo9682
paper_authors._id: 66ff5d6c661a30ae72a1b83c
paper_authors.name: Mu Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:16:17.446Z
paper_authors._id: 63b7b2c6bd2d153522821766
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
paper_authors.fullname: Mu Cai
paper_authors.type: user
paper_authors.user: mucai
paper_authors._id: 66ff5d6c661a30ae72a1b83d
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-04T09:15:57.098Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_title: Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short
    Videos
paper_summary: There has been growing sentiment recently that modern large multimodal models
  (LMMs) have addressed most of the key challenges related to short video
  comprehension. As a result, both academia and industry are gradually shifting
  their attention towards the more complex challenges posed by understanding
  long-form videos. However, is this really the case? Our studies indicate that
  LMMs still lack many fundamental reasoning capabilities even when dealing with
  short videos. We introduce Vinoground, a temporal counterfactual LMM evaluation
  benchmark encompassing 1000 short and natural video-caption pairs. We
  demonstrate that existing LMMs severely struggle to distinguish temporal
  differences between different actions and object transformations. For example,
  the best model GPT-4o only obtains ~50% on our text and video scores, showing a
  large gap compared to the human baseline of ~90%. All open-source multimodal
  models and CLIP-based models perform much worse, producing mostly random chance
  performance. Through this work, we shed light onto the fact that temporal
  reasoning in short videos is a problem yet to be fully solved. The dataset and
  evaluation code are available at https://vinoground.github.io.
paper_ai_summary: Vinoground is a new benchmark that reveals existing large multimodal models still lack temporal reasoning capabilities in short videos.
paper_ai_keywords: multimodal models LMMs temporal counterfactual video-caption pairs temporal reasoning short videos