Unleashing Hour-Scale Video Training for Long Video-Language
  Understanding
Recent long-form video-language understanding benchmarks have driven progress
in video large multimodal models (Video-LMMs). However, the scarcity of
well-annotated long videos has left the training of hour-long Video-LLMs
underexplored. To close this gap, we present VideoMarathon, a large-scale
hour-long video instruction-following dataset. This dataset includes around
9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60
minutes per video. Specifically, it contains 3.3M high-quality QA pairs,
spanning six fundamental topics: temporality, spatiality, object, action,
scene, and event. Compared to existing video instruction datasets,
VideoMarathon significantly extends training video durations up to 1 hour, and
supports 22 diverse tasks requiring both short- and long-term video
comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and
efficient Video-LMM for hour-scale video-language modeling. It enables
hour-long video training and inference at 1-FPS sampling by leveraging a memory
augmentation module, which adaptively integrates user question-relevant and
spatiotemporal-informative semantics from a cached full video context. In our
experiments, Hour-LLaVA achieves the best performance on multiple long
video-language benchmarks, demonstrating the high quality of the VideoMarathon
dataset and the superiority of the Hour-LLaVA model.
684240b70e7e14aac456fa59
Jingyang Lin
684240b70e7e14aac456fa5a
Jialian Wu
684240b70e7e14aac456fa5b
Ximeng Sun
684240b70e7e14aac456fa5c
Ze Wang
684240b70e7e14aac456fa5d
Jiang Liu
684240b70e7e14aac456fa5e
Yusheng Su
684240b70e7e14aac456fa5f
Xiaodong Yu
684240b70e7e14aac456fa60
Hao Chen
684240b70e7e14aac456fa61
Jiebo Luo
684240b70e7e14aac456fa62
Zicheng Liu
684240b70e7e14aac456fa63
Emad Barsoum
Unleashing Hour-Scale Video Training for Long Video-Language
  Understanding
Recent long-form video-language understanding benchmarks have driven progress
in video large multimodal models (Video-LMMs). However, the scarcity of
well-annotated long videos has left the training of hour-long Video-LLMs
underexplored. To close this gap, we present VideoMarathon, a large-scale
hour-long video instruction-following dataset. This dataset includes around
9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60
minutes per video. Specifically, it contains 3.3M high-quality QA pairs,
spanning six fundamental topics: temporality, spatiality, object, action,
scene, and event. Compared to existing video instruction datasets,
VideoMarathon significantly extends training video durations up to 1 hour, and
supports 22 diverse tasks requiring both short- and long-term video
comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and
efficient Video-LMM for hour-scale video-language modeling. It enables
hour-long video training and inference at 1-FPS sampling by leveraging a memory
augmentation module, which adaptively integrates user question-relevant and
spatiotemporal-informative semantics from a cached full video context. In our
experiments, Hour-LLaVA achieves the best performance on multiple long
video-language benchmarks, demonstrating the high quality of the VideoMarathon
dataset and the superiority of the Hour-LLaVA model.
Hour-LLaVA, a memory-augmented Video-LMM, achieves top performance on long video-language benchmarks using the VideoMarathon dataset, which extends video durations to one hour for comprehensive training and inference.
Video-LMMs VideoMarathon hour-long video instruction-following dataset QA pairs temporality spatiality object action scene event long-term video comprehension Hour-LLaVA memory augmentation module spatiotemporal-informative semantics cached full video context