ProTA: Probabilistic Token Aggregation for Text-Video Retrieval
<|reference_start|>ProTA: Probabilistic Token Aggregation for Text-Video Retrieval: Text-video retrieval aims to find the most relevant cross-modal samples for a given query. Recent methods focus on modeling the whole spatial-temporal relations. However, since video clips contain more diverse content than captions, the model aligning these asymmetric video-text pairs has a high risk of retrieving many false positive results. In this paper, we propose Probabilistic Token Aggregation (ProTA) to handle cross-modal interaction with content asymmetry. Specifically, we propose dual partial-related aggregation to disentangle and re-aggregate token representations in both low-dimension and high-dimension spaces. We propose token-based probabilistic alignment to generate token-level probabilistic representation and maintain the feature representation diversity. In addition, an adaptive contrastive loss is proposed to learn compact cross-modal distribution space. Based on extensive experiments, ProTA achieves significant improvements on MSR-VTT (50.9%), LSMDC (25.8%), and DiDeMo (47.2%).<|reference_end|>
@article{fang2024prota:,
    title={ProTA: Probabilistic Token Aggregation for Text-Video Retrieval},
    author={Han Fang, Xianghao Zang, Chao Ban, Zerun Feng, Lanxiang Zhou,
  Zhongjiang He, Yongxiang Li, Hao Sun},
    journal={arXiv preprint arXiv:2404.12216},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.12216},
    primaryClass={cs.CV}
}