VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models
<|reference_start|>VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models: Contrastive Language-Image Pre-training (CLIP) has been widely studied and applied in numerous applications. However, the emphasis on brief summary texts during pre-training prevents CLIP from understanding long descriptions. This issue is particularly acute regarding videos given that videos often contain abundant detailed contents. In this paper, we propose the VideoCLIP-XL (eXtra Length) model, which aims to unleash the long-description understanding capability of video CLIP models. Firstly, we establish an automatic data collection system and gather a large-scale VILD pre-training dataset with VIdeo and Long-Description pairs. Then, we propose Text-similarity-guided Primary Component Matching (TPCM) to better learn the distribution of feature space while expanding the long description capability. We also introduce two new tasks namely Detail-aware Description Ranking (DDR) and Hallucination-aware Description Ranking (HDR) for further understanding improvement. Finally, we construct a Long Video Description Ranking (LVDR) benchmark for evaluating the long-description capability more comprehensively. Extensive experimental results on widely-used text-video retrieval benchmarks with both short and long descriptions and our LVDR benchmark can fully demonstrate the effectiveness of our method.<|reference_end|>
@article{wang2024videoclip-xl:,
    title={VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP
  Models},
    author={Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, Lianwen Jin},
    journal={arXiv preprint arXiv:2410.00741},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.00741},
    primaryClass={cs.CL cs.CV cs.MM}
}