TemporalBench: Benchmarking Fine-grained Temporal Understanding for
  Multimodal Video Models
Understanding fine-grained temporal dynamics is crucial for multimodal video
comprehension and generation. Due to the lack of fine-grained temporal
annotations, existing video benchmarks mostly resemble static image benchmarks
and are incompetent at evaluating models for temporal understanding. In this
paper, we introduce TemporalBench, a new benchmark dedicated to evaluating
fine-grained temporal understanding in videos. TemporalBench consists of ~10K
video question-answer pairs, derived from ~2K high-quality human annotations
detailing the temporal dynamics in video clips. As a result, our benchmark
provides a unique testbed for evaluating various temporal understanding and
reasoning abilities such as action frequency, motion magnitude, event order,
etc. Moreover, it enables evaluations on various tasks like both video question
answering and captioning, both short and long video understanding, as well as
different models such as multimodal video embedding models and text generation
models. Results show that state-of-the-art models like GPT-4o achieve only
38.5% question answering accuracy on TemporalBench, demonstrating a significant
gap (~30%) between humans and AI in temporal understanding. Furthermore, we
notice a critical pitfall for multi-choice QA where LLMs can detect the subtle
changes in negative captions and find a centralized description as a cue for
its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such
bias. We hope that TemporalBench can foster research on improving models'
temporal reasoning capabilities. Both dataset and evaluation code will be made
available.
670de52bab8a8a8e3b942151
Mu Cai
admin_assigned
2024-10-15T14:09:25.460Z
63b7b2c6bd2d153522821766
https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
Mu Cai
user
mucai
670de52bab8a8a8e3b942152
Reuben Tan
670de52bab8a8a8e3b942153
Jianrui Zhang
claimed_verified
2024-11-15T09:43:17.298Z
6508b164abdde5290e5e4939
https://cdn-avatars.huggingface.co/v1/production/uploads/6508b164abdde5290e5e4939/lQgAs3BHwCyI7Go1QA62m.jpeg
Harris Zhang
user
HanSolo9682
670de52bab8a8a8e3b942154
Bocheng Zou
claimed_verified
2024-10-15T08:07:52.341Z
6489eb3c44cfcffe8f5918e3
/avatars/4edffaa045cc736e2edbf858fb6ef7c8.svg
Bocheng Zou
user
BochengZou
670de52bab8a8a8e3b942155
Kai Zhang
admin_assigned
2024-10-15T14:10:19.304Z
63e0a50242591dda0b9dca5c
https://cdn-avatars.huggingface.co/v1/production/uploads/63e0a50242591dda0b9dca5c/c7cBPEBWQDFYimfGnO_SI.png
Kai Zhang
user
drogozhang
670de52bab8a8a8e3b942156
Feng Yao
claimed_verified
2024-10-15T08:07:54.360Z
64c8b2c5c547ed5243e14a6e
/avatars/96d4a9010f96001c8cff235915926390.svg
Feng Yao
user
fengyao1909
670de52bab8a8a8e3b942157
Fangrui Zhu
670de52bab8a8a8e3b942158
Jing Gu
admin_assigned
2024-10-15T14:10:56.430Z
634781fae5c0717e6739bc59
/avatars/2b6d2d0625741b28f326459acad55b69.svg
Jing Gu
user
jinggu
670de52bab8a8a8e3b942159
Yiwu Zhong
admin_assigned
2024-10-15T14:12:43.229Z
62b4fe7ab25cb80fcf2ffd66
/avatars/22f6a05cdbf4224d29ec9259c9fdd7a4.svg
Yiwu Zhong
user
YiwuZhong
670de52bab8a8a8e3b94215a
Yuzhang Shang
admin_assigned
2024-10-15T14:12:48.439Z
63a740aec46291f273a95b4c
/avatars/8c19e639e39f93083ab624747f9fd0b3.svg
Yuzhang Shang
user
yuzhang
670de52bab8a8a8e3b94215b
Yao Dou
admin_assigned
2024-10-15T14:12:54.131Z
63519eb8a9a9ae182206ff02
/avatars/c310ac17d87bd4d892ff29caad2673dd.svg
Yao Dou
user
douy
670de52bab8a8a8e3b94215c
Jaden Park
admin_assigned
2024-10-15T14:13:04.199Z
66421caf1208117512556149
/avatars/71cc7eff6c914e5369697dc6f4daa0bb.svg
pak
user
jadenpak
670de52bab8a8a8e3b94215d
Jianfeng Gao
admin_assigned
2024-10-15T14:13:10.670Z
641904caf9d6f1d772ec7af7
/avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
Jianfeng Gao
user
wyngjf
670de52bab8a8a8e3b94215e
Yong Jae Lee
admin_assigned
2024-10-15T14:13:15.639Z
649f41ee70a478f8b36b2984
/avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
Yong Jae Lee
user
yjlee0222
670de52bab8a8a8e3b94215f
Jianwei Yang
admin_assigned
2024-10-15T14:13:36.677Z
6125df7f25027fb1ea9c7a41
https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
Jianwei Yang
user
jw2yang
TemporalBench: Benchmarking Fine-grained Temporal Understanding for
  Multimodal Video Models
Understanding fine-grained temporal dynamics is crucial for multimodal video
comprehension and generation. Due to the lack of fine-grained temporal
annotations, existing video benchmarks mostly resemble static image benchmarks
and are incompetent at evaluating models for temporal understanding. In this
paper, we introduce TemporalBench, a new benchmark dedicated to evaluating
fine-grained temporal understanding in videos. TemporalBench consists of ~10K
video question-answer pairs, derived from ~2K high-quality human annotations
detailing the temporal dynamics in video clips. As a result, our benchmark
provides a unique testbed for evaluating various temporal understanding and
reasoning abilities such as action frequency, motion magnitude, event order,
etc. Moreover, it enables evaluations on various tasks like both video question
answering and captioning, both short and long video understanding, as well as
different models such as multimodal video embedding models and text generation
models. Results show that state-of-the-art models like GPT-4o achieve only
38.5% question answering accuracy on TemporalBench, demonstrating a significant
gap (~30%) between humans and AI in temporal understanding. Furthermore, we
notice a critical pitfall for multi-choice QA where LLMs can detect the subtle
changes in negative captions and find a centralized description as a cue for
its prediction, where we propose Multiple Binary Accuracy (MBA) to correct such
bias. We hope that TemporalBench can foster research on improving models'
temporal reasoning capabilities. Both dataset and evaluation code will be made
available.
TemporalBench is a new benchmark for evaluating fine-grained temporal understanding in videos, highlighting gaps in model performance and proposing a method to address biases in multi-choice QA.
fine-grained temporal understanding TemporalBench video question-answer pairs action frequency motion magnitude event order video question answering video captioning multimodal video embedding models text generation models Multiple Binary Accuracy (MBA)