title: Unleashing Hour-Scale Video Training for Long Video-Language
    Understanding
summary: Recent long-form video-language understanding benchmarks have driven progress
  in video large multimodal models (Video-LMMs). However, the scarcity of
  well-annotated long videos has left the training of hour-long Video-LLMs
  underexplored. To close this gap, we present VideoMarathon, a large-scale
  hour-long video instruction-following dataset. This dataset includes around
  9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60
  minutes per video. Specifically, it contains 3.3M high-quality QA pairs,
  spanning six fundamental topics: temporality, spatiality, object, action,
  scene, and event. Compared to existing video instruction datasets,
  VideoMarathon significantly extends training video durations up to 1 hour, and
  supports 22 diverse tasks requiring both short- and long-term video
  comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and
  efficient Video-LMM for hour-scale video-language modeling. It enables
  hour-long video training and inference at 1-FPS sampling by leveraging a memory
  augmentation module, which adaptively integrates user question-relevant and
  spatiotemporal-informative semantics from a cached full video context. In our
  experiments, Hour-LLaVA achieves the best performance on multiple long
  video-language benchmarks, demonstrating the high quality of the VideoMarathon
  dataset and the superiority of the Hour-LLaVA model.
paper_authors._id: 684240b70e7e14aac456fa59
paper_authors.name: Jingyang Lin
paper_authors._id: 684240b70e7e14aac456fa5a
paper_authors.name: Jialian Wu
paper_authors._id: 684240b70e7e14aac456fa5b
paper_authors.name: Ximeng Sun
paper_authors._id: 684240b70e7e14aac456fa5c
paper_authors.name: Ze Wang
paper_authors._id: 684240b70e7e14aac456fa5d
paper_authors.name: Jiang Liu
paper_authors._id: 684240b70e7e14aac456fa5e
paper_authors.name: Yusheng Su
paper_authors._id: 684240b70e7e14aac456fa5f
paper_authors.name: Xiaodong Yu
paper_authors._id: 684240b70e7e14aac456fa60
paper_authors.name: Hao Chen
paper_authors._id: 684240b70e7e14aac456fa61
paper_authors.name: Jiebo Luo
paper_authors._id: 684240b70e7e14aac456fa62
paper_authors.name: Zicheng Liu
paper_authors._id: 684240b70e7e14aac456fa63
paper_authors.name: Emad Barsoum
paper_title: Unleashing Hour-Scale Video Training for Long Video-Language
    Understanding
paper_summary: Recent long-form video-language understanding benchmarks have driven progress
  in video large multimodal models (Video-LMMs). However, the scarcity of
  well-annotated long videos has left the training of hour-long Video-LLMs
  underexplored. To close this gap, we present VideoMarathon, a large-scale
  hour-long video instruction-following dataset. This dataset includes around
  9,700 hours of long videos sourced from diverse domains, ranging from 3 to 60
  minutes per video. Specifically, it contains 3.3M high-quality QA pairs,
  spanning six fundamental topics: temporality, spatiality, object, action,
  scene, and event. Compared to existing video instruction datasets,
  VideoMarathon significantly extends training video durations up to 1 hour, and
  supports 22 diverse tasks requiring both short- and long-term video
  comprehension. Building on VideoMarathon, we propose Hour-LLaVA, a powerful and
  efficient Video-LMM for hour-scale video-language modeling. It enables
  hour-long video training and inference at 1-FPS sampling by leveraging a memory
  augmentation module, which adaptively integrates user question-relevant and
  spatiotemporal-informative semantics from a cached full video context. In our
  experiments, Hour-LLaVA achieves the best performance on multiple long
  video-language benchmarks, demonstrating the high quality of the VideoMarathon
  dataset and the superiority of the Hour-LLaVA model.
paper_ai_summary: Hour-LLaVA, a memory-augmented Video-LMM, achieves top performance on long video-language benchmarks using the VideoMarathon dataset, which extends video durations to one hour for comprehensive training and inference.
paper_ai_keywords: Video-LMMs VideoMarathon hour-long video instruction-following dataset QA pairs temporality spatiality object action scene event long-term video comprehension Hour-LLaVA memory augmentation module spatiotemporal-informative semantics cached full video context