ViLCo-Bench: VIdeo Language COntinual learning Benchmark
<|reference_start|>ViLCo-Bench: VIdeo Language COntinual learning Benchmark: Video language continual learning involves continuously adapting to information from video and text inputs, enhancing a model's ability to handle new tasks while retaining prior knowledge. This field is a relatively under-explored area, and establishing appropriate datasets is crucial for facilitating communication and research in this field. In this study, we present the first dedicated benchmark, ViLCo-Bench, designed to evaluate continual learning models across a range of video-text tasks. The dataset comprises ten-minute-long videos and corresponding language queries collected from publicly available datasets. Additionally, we introduce a novel memory-efficient framework that incorporates self-supervised learning and mimics long-term and short-term memory effects. This framework addresses challenges including memory complexity from long video clips, natural language complexity from open queries, and text-video misalignment. We posit that ViLCo-Bench, with greater complexity compared to existing continual learning benchmarks, would serve as a critical tool for exploring the video-language domain, extending beyond conventional class-incremental tasks, and addressing complex and limited annotation issues. The curated data, evaluations, and our novel method are available at https://github.com/cruiseresearchgroup/ViLCo .<|reference_end|>
@article{tang2024vilco-bench:,
    title={ViLCo-Bench: VIdeo Language COntinual learning Benchmark},
    author={Tianqi Tang, Shohreh Deldari, Hao Xue, Celso De Melo, Flora D. Salim},
    journal={arXiv preprint arXiv:2406.13123},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.13123},
    primaryClass={cs.AI cs.CV}
}