TEMPURA: Temporal Event Masked Prediction and Understanding for
  Reasoning in Action
Understanding causal event relationships and achieving fine-grained temporal
grounding in videos remain challenging for vision-language models. Existing
methods either compress video tokens to reduce temporal resolution, or treat
videos as unsegmented streams, which obscures fine-grained event boundaries and
limits the modeling of causal dependencies. We propose TEMPURA (Temporal Event
Masked Prediction and Understanding for Reasoning in Action), a two-stage
training framework that enhances video temporal understanding. TEMPURA first
applies masked event prediction reasoning to reconstruct missing events and
generate step-by-step causal explanations from dense event annotations, drawing
inspiration from effective infilling techniques. TEMPURA then learns to perform
video segmentation and dense captioning to decompose videos into
non-overlapping events with detailed, timestamp-aligned descriptions. We train
TEMPURA on VER, a large-scale dataset curated by us that comprises 1M training
instances and 500K videos with temporally aligned event descriptions and
structured reasoning steps. Experiments on temporal grounding and highlight
detection benchmarks demonstrate that TEMPURA outperforms strong baseline
models, confirming that integrating causal reasoning with fine-grained temporal
segmentation leads to improved video understanding.
6819814653612b577df718e7
Jen-Hao Cheng
claimed_verified
2025-05-06T08:33:15.728Z
65cd4d6256671dee8ee46392
https://cdn-avatars.huggingface.co/v1/production/uploads/65cd4d6256671dee8ee46392/SH30XVQnGiYqYQIeDk3na.jpeg
Jen-Hao (Andy) Cheng
user
andaba
6819814653612b577df718e8
Vivian Wang
6819814653612b577df718e9
Huayu Wang
6819814653612b577df718ea
Huapeng Zhou
6819814653612b577df718eb
Yi-Hao Peng
6819814653612b577df718ec
Hou-I Liu
6819814653612b577df718ed
Hsiang-Wei Huang
admin_assigned
2025-05-06T09:13:19.727Z
647e4e8da49bffab5d72fbe0
/avatars/c5fb00019c7cea23fe3351ecb1e43195.svg
Hsiang-Wei Huang
user
hsiangwei0903
6819814653612b577df718ee
Kuang-Ming Chen
6819814653612b577df718ef
Cheng-Yen Yang
6819814653612b577df718f0
Wenhao Chai
admin_assigned
2025-05-06T09:13:37.240Z
637c7503fe115289cfecbe6b
https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
Wenhao Chai
user
wchai
6819814653612b577df718f1
Yi-Ling Chen
admin_assigned
2025-05-06T09:13:43.313Z
65f8cb651e0c65c13a2b906a
/avatars/ffc8ac8f29ab1a3142fe5fab1b2302ca.svg
Yi-Ling Chen
user
yilche
6819814653612b577df718f2
Vibhav Vineet
admin_assigned
2025-05-06T09:13:49.482Z
63c8527becdb7c9fdd9cacc6
/avatars/c8a3f5e1e5159ae5ead41bd9fc2b9b34.svg
Vibhav Vineet
user
vibhav-vineet
6819814653612b577df718f3
Qin Cai
6819814653612b577df718f4
Jenq-Neng Hwang
TEMPURA: Temporal Event Masked Prediction and Understanding for
  Reasoning in Action
Understanding causal event relationships and achieving fine-grained temporal
grounding in videos remain challenging for vision-language models. Existing
methods either compress video tokens to reduce temporal resolution, or treat
videos as unsegmented streams, which obscures fine-grained event boundaries and
limits the modeling of causal dependencies. We propose TEMPURA (Temporal Event
Masked Prediction and Understanding for Reasoning in Action), a two-stage
training framework that enhances video temporal understanding. TEMPURA first
applies masked event prediction reasoning to reconstruct missing events and
generate step-by-step causal explanations from dense event annotations, drawing
inspiration from effective infilling techniques. TEMPURA then learns to perform
video segmentation and dense captioning to decompose videos into
non-overlapping events with detailed, timestamp-aligned descriptions. We train
TEMPURA on VER, a large-scale dataset curated by us that comprises 1M training
instances and 500K videos with temporally aligned event descriptions and
structured reasoning steps. Experiments on temporal grounding and highlight
detection benchmarks demonstrate that TEMPURA outperforms strong baseline
models, confirming that integrating causal reasoning with fine-grained temporal
segmentation leads to improved video understanding.
TEMPURA, a two-stage training framework, enhances video temporal understanding by reconstructing missing events, generating causal explanations, and performing dense video segmentation and captioning using a large-scale dataset.
TEMPURA masked event prediction causal explanations dense event annotations infilling techniques video segmentation dense captioning temporally aligned event descriptions structured reasoning steps temporal grounding highlight detection