VidLA: Video-Language Alignment at Scale
In this paper, we propose VidLA, an approach for video-language alignment at
scale. There are two major limitations of previous video-language alignment
approaches. First, they do not capture both short-range and long-range temporal
dependencies and typically employ complex hierarchical deep network
architectures that are hard to integrate with existing pretrained image-text
foundation models. To effectively address this limitation, we instead keep the
network architecture simple and use a set of data tokens that operate at
different temporal resolutions in a hierarchical manner, accounting for the
temporally hierarchical nature of videos. By employing a simple two-tower
architecture, we are able to initialize our video-language model with
pretrained image-text foundation models, thereby boosting the final
performance. Second, existing video-language alignment works struggle due to
the lack of semantically aligned large-scale training data. To overcome it, we
leverage recent LLMs to curate the largest video-language dataset to date with
better visual grounding. Furthermore, unlike existing video-text datasets which
only contain short clips, our dataset is enriched with video clips of varying
durations to aid our temporally hierarchical data tokens in extracting better
representations at varying temporal scales. Overall, empirical results show
that our proposed approach surpasses state-of-the-art methods on multiple
retrieval benchmarks, especially on longer videos, and performs competitively
on classification benchmarks.
66010210b1e509e1e4b15bf7
Mamshad Nayeem Rizve
admin_assigned
2024-03-25T10:44:55.192Z
6419e791e8a6183a8cada336
/avatars/4857607039bea4fa791643df07b1e998.svg
Mamshad Nayeem Rizve
user
nayeemrizve
66010210b1e509e1e4b15bf8
Fan Fei
66010210b1e509e1e4b15bf9
Jayakrishnan Unnikrishnan
66010210b1e509e1e4b15bfa
Son Tran
66010210b1e509e1e4b15bfb
Benjamin Z. Yao
66010210b1e509e1e4b15bfc
Belinda Zeng
66010210b1e509e1e4b15bfd
Mubarak Shah
66010210b1e509e1e4b15bfe
Trishul Chilimbi
VidLA: Video-Language Alignment at Scale
In this paper, we propose VidLA, an approach for video-language alignment at
scale. There are two major limitations of previous video-language alignment
approaches. First, they do not capture both short-range and long-range temporal
dependencies and typically employ complex hierarchical deep network
architectures that are hard to integrate with existing pretrained image-text
foundation models. To effectively address this limitation, we instead keep the
network architecture simple and use a set of data tokens that operate at
different temporal resolutions in a hierarchical manner, accounting for the
temporally hierarchical nature of videos. By employing a simple two-tower
architecture, we are able to initialize our video-language model with
pretrained image-text foundation models, thereby boosting the final
performance. Second, existing video-language alignment works struggle due to
the lack of semantically aligned large-scale training data. To overcome it, we
leverage recent LLMs to curate the largest video-language dataset to date with
better visual grounding. Furthermore, unlike existing video-text datasets which
only contain short clips, our dataset is enriched with video clips of varying
durations to aid our temporally hierarchical data tokens in extracting better
representations at varying temporal scales. Overall, empirical results show
that our proposed approach surpasses state-of-the-art methods on multiple
retrieval benchmarks, especially on longer videos, and performs competitively
on classification benchmarks.
VidLA, a simple yet effective video-language alignment approach, uses temporally hierarchical data tokens and a two-tower architecture to enhance performance with large, semantically aligned datasets and surpass state-of-the-art methods.
video-language alignment temporal dependencies hierarchical network architectures data tokens two-tower architecture pretrained image-text foundation models semantically aligned large-scale training data LLMs temporally hierarchical data tokens retrieval benchmarks classification benchmarks