EA-VTR: Event-Aware Video-Text Retrieval
<|reference_start|>EA-VTR: Event-Aware Video-Text Retrieval: Understanding the content of events occurring in the video and their inherent temporal logic is crucial for video-text retrieval. However, web-crawled pre-training datasets often lack sufficient event information, and the widely adopted video-level cross-modal contrastive learning also struggles to capture detailed and complex video-text event alignment. To address these challenges, we make improvements from both data and model perspectives. In terms of pre-training data, we focus on supplementing the missing specific event content and event temporal transitions with the proposed event augmentation strategies. Based on the event-augmented data, we construct a novel Event-Aware Video-Text Retrieval model, ie, EA-VTR, which achieves powerful video-text retrieval ability through superior video event awareness. EA-VTR can efficiently encode frame-level and video-level visual representations simultaneously, enabling detailed event content and complex event temporal cross-modal alignment, ultimately enhancing the comprehensive understanding of video events. Our method not only significantly outperforms existing approaches on multiple datasets for Text-to-Video Retrieval and Video Action Recognition tasks, but also demonstrates superior event content perceive ability on Multi-event Video-Text Retrieval and Video Moment Retrieval tasks, as well as outstanding event temporal logic understanding ability on Test of Time task.<|reference_end|>
@article{ma2024ea-vtr:,
    title={EA-VTR: Event-Aware Video-Text Retrieval},
    author={Zongyang Ma, Ziqi Zhang, Yuxin Chen, Zhongang Qi, Chunfeng Yuan, Bing
  Li, Yingmin Luo, Xu Li, Xiaojuan Qi, Ying Shan, Weiming Hu},
    journal={arXiv preprint arXiv:2407.07478},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.07478},
    primaryClass={cs.CV}
}