title: TemporalBench: Benchmarking Fine-grained Temporal Understanding for
    Multimodal Video Models
summary: Understanding fine-grained temporal dynamics is crucial for multimodal video
  comprehension and generation. Due to the lack of fine-grained temporal
  annotations, existing video benchmarks mostly resemble static image benchmarks
  and are incompetent at evaluating models for temporal understanding. In this
  paper, we introduce TemporalBench, a new benchmark dedicated to evaluating
  fine-grained temporal understanding in videos. TemporalBench consists of ~10K
  video question-answer pairs, derived from ~2K high-quality human annotations
  detailing the temporal dynamics in video clips. As a result, our benchmark
  provides a unique testbed for evaluating various temporal understanding and
  reasoning abilities such as action frequency, motion magnitude, event order,
  etc. Moreover, it enables evaluations on various tasks like both video question
  answering and captioning, both short and long video understanding, as well as
  different models such as multimodal video embedding models and text generation
  models. Results show that state-of-the-art models like GPT-4o achieve only
  38.5% question answering accuracy on TemporalBench, demonstrating a significant
  gap (~30%) between humans and AI in temporal understanding. Furthermore, we
  notice a critical pitfall for multi-choice QA where LLMs can detect the subtle
  changes in negative captions and find a centralized description as a cue for
  its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such
  bias. We hope that TemporalBench can foster research on improving models'
  temporal reasoning capabilities. Both dataset and evaluation code will be made
  available.
paper_authors._id: 670de52bab8a8a8e3b942151
paper_authors.name: Mu Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:09:25.460Z
paper_authors._id: 63b7b2c6bd2d153522821766
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
paper_authors.fullname: Mu Cai
paper_authors.type: user
paper_authors.user: mucai
paper_authors._id: 670de52bab8a8a8e3b942152
paper_authors.name: Reuben Tan
paper_authors._id: 670de52bab8a8a8e3b942153
paper_authors.name: Jianrui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:43:17.298Z
paper_authors._id: 6508b164abdde5290e5e4939
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6508b164abdde5290e5e4939/lQgAs3BHwCyI7Go1QA62m.jpeg
paper_authors.fullname: Harris Zhang
paper_authors.type: user
paper_authors.user: HanSolo9682
paper_authors._id: 670de52bab8a8a8e3b942154
paper_authors.name: Bocheng Zou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T08:07:52.341Z
paper_authors._id: 6489eb3c44cfcffe8f5918e3
paper_authors.avatarUrl: /avatars/4edffaa045cc736e2edbf858fb6ef7c8.svg
paper_authors.fullname: Bocheng Zou
paper_authors.type: user
paper_authors.user: BochengZou
paper_authors._id: 670de52bab8a8a8e3b942155
paper_authors.name: Kai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:10:19.304Z
paper_authors._id: 63e0a50242591dda0b9dca5c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: drogozhang
paper_authors._id: 670de52bab8a8a8e3b942156
paper_authors.name: Feng Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T08:07:54.360Z
paper_authors._id: 64c8b2c5c547ed5243e14a6e
paper_authors.avatarUrl: /avatars/96d4a9010f96001c8cff235915926390.svg
paper_authors.fullname: Feng Yao
paper_authors.type: user
paper_authors.user: fengyao1909
paper_authors._id: 670de52bab8a8a8e3b942157
paper_authors.name: Fangrui Zhu
paper_authors._id: 670de52bab8a8a8e3b942158
paper_authors.name: Jing Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:10:56.430Z
paper_authors._id: 634781fae5c0717e6739bc59
paper_authors.avatarUrl: /avatars/2b6d2d0625741b28f326459acad55b69.svg
paper_authors.fullname: Jing Gu
paper_authors.type: user
paper_authors.user: jinggu
paper_authors._id: 670de52bab8a8a8e3b942159
paper_authors.name: Yiwu Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:12:43.229Z
paper_authors._id: 62b4fe7ab25cb80fcf2ffd66
paper_authors.avatarUrl: /avatars/22f6a05cdbf4224d29ec9259c9fdd7a4.svg
paper_authors.fullname: Yiwu Zhong
paper_authors.type: user
paper_authors.user: YiwuZhong
paper_authors._id: 670de52bab8a8a8e3b94215a
paper_authors.name: Yuzhang Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:12:48.439Z
paper_authors._id: 63a740aec46291f273a95b4c
paper_authors.avatarUrl: /avatars/8c19e639e39f93083ab624747f9fd0b3.svg
paper_authors.fullname: Yuzhang Shang
paper_authors.type: user
paper_authors.user: yuzhang
paper_authors._id: 670de52bab8a8a8e3b94215b
paper_authors.name: Yao Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:12:54.131Z
paper_authors._id: 63519eb8a9a9ae182206ff02
paper_authors.avatarUrl: /avatars/c310ac17d87bd4d892ff29caad2673dd.svg
paper_authors.fullname: Yao Dou
paper_authors.type: user
paper_authors.user: douy
paper_authors._id: 670de52bab8a8a8e3b94215c
paper_authors.name: Jaden Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:13:04.199Z
paper_authors._id: 66421caf1208117512556149
paper_authors.avatarUrl: /avatars/71cc7eff6c914e5369697dc6f4daa0bb.svg
paper_authors.fullname: pak
paper_authors.type: user
paper_authors.user: jadenpak
paper_authors._id: 670de52bab8a8a8e3b94215d
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:13:10.670Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 670de52bab8a8a8e3b94215e
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:13:15.639Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_authors._id: 670de52bab8a8a8e3b94215f
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-15T14:13:36.677Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_title: TemporalBench: Benchmarking Fine-grained Temporal Understanding for
    Multimodal Video Models
paper_summary: Understanding fine-grained temporal dynamics is crucial for multimodal video
  comprehension and generation. Due to the lack of fine-grained temporal
  annotations, existing video benchmarks mostly resemble static image benchmarks
  and are incompetent at evaluating models for temporal understanding. In this
  paper, we introduce TemporalBench, a new benchmark dedicated to evaluating
  fine-grained temporal understanding in videos. TemporalBench consists of ~10K
  video question-answer pairs, derived from ~2K high-quality human annotations
  detailing the temporal dynamics in video clips. As a result, our benchmark
  provides a unique testbed for evaluating various temporal understanding and
  reasoning abilities such as action frequency, motion magnitude, event order,
  etc. Moreover, it enables evaluations on various tasks like both video question
  answering and captioning, both short and long video understanding, as well as
  different models such as multimodal video embedding models and text generation
  models. Results show that state-of-the-art models like GPT-4o achieve only
  38.5% question answering accuracy on TemporalBench, demonstrating a significant
  gap (~30%) between humans and AI in temporal understanding. Furthermore, we
  notice a critical pitfall for multi-choice QA where LLMs can detect the subtle
  changes in negative captions and find a centralized description as a cue for
  its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such
  bias. We hope that TemporalBench can foster research on improving models'
  temporal reasoning capabilities. Both dataset and evaluation code will be made
  available.
paper_ai_keywords: Video question-answer pairs Temporal understanding Multimodal video comprehension Video generation Fine-grained temporal dynamics Multimodal video embedding Text generation TemporalBench GPT-4 Multiple Binary Binary (MBA) Temporal reasoning