title: VISTA: Enhancing Long-Duration and High-Resolution Video Understanding
    by Video Spatiotemporal Augmentation
summary: Current large multimodal models (LMMs) face significant challenges in
  processing and comprehending long-duration or high-resolution videos, which is
  mainly due to the lack of high-quality datasets. To address this issue from a
  data-centric perspective, we propose VISTA, a simple yet effective Video
  Spatiotemporal Augmentation framework that synthesizes long-duration and
  high-resolution video instruction-following pairs from existing video-caption
  datasets. VISTA spatially and temporally combines videos to create new
  synthetic videos with extended durations and enhanced resolutions, and
  subsequently produces question-answer pairs pertaining to these newly
  synthesized videos. Based on this paradigm, we develop seven video augmentation
  methods and curate VISTA-400K, a video instruction-following dataset aimed at
  enhancing long-duration and high-resolution video understanding. Finetuning
  various video LMMs on our data resulted in an average improvement of 3.3%
  across four challenging benchmarks for long-video understanding. Furthermore,
  we introduce the first comprehensive high-resolution video understanding
  benchmark HRVideoBench, on which our finetuned models achieve a 6.5%
  performance gain. These results highlight the effectiveness of our framework.
paper_authors._id: 674e99abe91289226e0428b4
paper_authors.name: Weiming Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:14:38.615Z
paper_authors._id: 64405a9d518271b0d1beea38
paper_authors.avatarUrl: /avatars/b702474588fd7090773320422417a582.svg
paper_authors.fullname: Weiming Ren
paper_authors.type: user
paper_authors.user: wren93
paper_authors._id: 674e99abe91289226e0428b5
paper_authors.name: Huan Yang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-03T05:39:57.190Z
paper_authors._id: 639005861c5a6237272557d1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1670382915881-noauth.jpeg
paper_authors.fullname: Huan Yang
paper_authors.type: user
paper_authors.user: hyang0511
paper_authors._id: 674e99abe91289226e0428b6
paper_authors.name: Jie Min
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:29.622Z
paper_authors._id: 6551419b35d81e515311780d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/X0jRpQ3wEhLD6pfKlXIQe.jpeg
paper_authors.fullname: Jie Min
paper_authors.type: user
paper_authors.user: Stefan8
paper_authors._id: 674e99abe91289226e0428b7
paper_authors.name: Cong Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-05T09:28:27.503Z
paper_authors._id: 64f8e358766ff9f3d2b0de84
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f8e358766ff9f3d2b0de84/R2P1YG-mRBh7TU9wkjGGk.jpeg
paper_authors.fullname: Cong Wei
paper_authors.type: user
paper_authors.user: lim142857
paper_authors._id: 674e99abe91289226e0428b8
paper_authors.name: Wenhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T15:22:40.605Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: VISTA: Enhancing Long-Duration and High-Resolution Video Understanding
    by Video Spatiotemporal Augmentation
paper_summary: Current large multimodal models (LMMs) face significant challenges in
  processing and comprehending long-duration or high-resolution videos, which is
  mainly due to the lack of high-quality datasets. To address this issue from a
  data-centric perspective, we propose VISTA, a simple yet effective Video
  Spatiotemporal Augmentation framework that synthesizes long-duration and
  high-resolution video instruction-following pairs from existing video-caption
  datasets. VISTA spatially and temporally combines videos to create new
  synthetic videos with extended durations and enhanced resolutions, and
  subsequently produces question-answer pairs pertaining to these newly
  synthesized videos. Based on this paradigm, we develop seven video augmentation
  methods and curate VISTA-400K, a video instruction-following dataset aimed at
  enhancing long-duration and high-resolution video understanding. Finetuning
  various video LMMs on our data resulted in an average improvement of 3.3%
  across four challenging benchmarks for long-video understanding. Furthermore,
  we introduce the first comprehensive high-resolution video understanding
  benchmark HRVideoBench, on which our finetuned models achieve a 6.5%
  performance gain. These results highlight the effectiveness of our framework.
paper_ai_keywords: Video Spatiotemporal Augmentation Instruction-following pairs High-quality datasets Synthetic videos Extended durations Explicit resolutions Question-answer pairs Video augmentation methods Video instruction-following dataset Long-duration video understanding High-resolution video understanding Finetuning Video LMMs Benchmarks Performance gain HRVideoBench