title: VidLA: Video-Language Alignment at Scale
summary: In this paper, we propose VidLA, an approach for video-language alignment at
  scale. There are two major limitations of previous video-language alignment
  approaches. First, they do not capture both short-range and long-range temporal
  dependencies and typically employ complex hierarchical deep network
  architectures that are hard to integrate with existing pretrained image-text
  foundation models. To effectively address this limitation, we instead keep the
  network architecture simple and use a set of data tokens that operate at
  different temporal resolutions in a hierarchical manner, accounting for the
  temporally hierarchical nature of videos. By employing a simple two-tower
  architecture, we are able to initialize our video-language model with
  pretrained image-text foundation models, thereby boosting the final
  performance. Second, existing video-language alignment works struggle due to
  the lack of semantically aligned large-scale training data. To overcome it, we
  leverage recent LLMs to curate the largest video-language dataset to date with
  better visual grounding. Furthermore, unlike existing video-text datasets which
  only contain short clips, our dataset is enriched with video clips of varying
  durations to aid our temporally hierarchical data tokens in extracting better
  representations at varying temporal scales. Overall, empirical results show
  that our proposed approach surpasses state-of-the-art methods on multiple
  retrieval benchmarks, especially on longer videos, and performs competitively
  on classification benchmarks.
paper_authors._id: 66010210b1e509e1e4b15bf7
paper_authors.name: Mamshad Nayeem Rizve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-25T10:44:55.192Z
paper_authors._id: 6419e791e8a6183a8cada336
paper_authors.avatarUrl: /avatars/4857607039bea4fa791643df07b1e998.svg
paper_authors.fullname: Mamshad Nayeem Rizve
paper_authors.type: user
paper_authors.user: nayeemrizve
paper_authors._id: 66010210b1e509e1e4b15bf8
paper_authors.name: Fan Fei
paper_authors._id: 66010210b1e509e1e4b15bf9
paper_authors.name: Jayakrishnan Unnikrishnan
paper_authors._id: 66010210b1e509e1e4b15bfa
paper_authors.name: Son Tran
paper_authors._id: 66010210b1e509e1e4b15bfb
paper_authors.name: Benjamin Z. Yao
paper_authors._id: 66010210b1e509e1e4b15bfc
paper_authors.name: Belinda Zeng
paper_authors._id: 66010210b1e509e1e4b15bfd
paper_authors.name: Mubarak Shah
paper_authors._id: 66010210b1e509e1e4b15bfe
paper_authors.name: Trishul Chilimbi
paper_title: VidLA: Video-Language Alignment at Scale
paper_summary: In this paper, we propose VidLA, an approach for video-language alignment at
  scale. There are two major limitations of previous video-language alignment
  approaches. First, they do not capture both short-range and long-range temporal
  dependencies and typically employ complex hierarchical deep network
  architectures that are hard to integrate with existing pretrained image-text
  foundation models. To effectively address this limitation, we instead keep the
  network architecture simple and use a set of data tokens that operate at
  different temporal resolutions in a hierarchical manner, accounting for the
  temporally hierarchical nature of videos. By employing a simple two-tower
  architecture, we are able to initialize our video-language model with
  pretrained image-text foundation models, thereby boosting the final
  performance. Second, existing video-language alignment works struggle due to
  the lack of semantically aligned large-scale training data. To overcome it, we
  leverage recent LLMs to curate the largest video-language dataset to date with
  better visual grounding. Furthermore, unlike existing video-text datasets which
  only contain short clips, our dataset is enriched with video clips of varying
  durations to aid our temporally hierarchical data tokens in extracting better
  representations at varying temporal scales. Overall, empirical results show
  that our proposed approach surpasses state-of-the-art methods on multiple
  retrieval benchmarks, especially on longer videos, and performs competitively
  on classification benchmarks.
paper_ai_summary: VidLA, a simple yet effective video-language alignment approach, uses temporally hierarchical data tokens and a two-tower architecture to enhance performance with large, semantically aligned datasets and surpass state-of-the-art methods.
paper_ai_keywords: video-language alignment temporal dependencies hierarchical network architectures data tokens two-tower architecture pretrained image-text foundation models semantically aligned large-scale training data LLMs temporally hierarchical data tokens retrieval benchmarks classification benchmarks