FILS: Self-Supervised Video Feature Prediction In Semantic Language Space
<|reference_start|>FILS: Self-Supervised Video Feature Prediction In Semantic Language Space: This paper demonstrates a self-supervised approach for learning semantic video representations. Recent vision studies show that a masking strategy for vision and natural language supervision has contributed to developing transferable visual pretraining. Our goal is to achieve a more semantic video representation by leveraging the text related to the video content during the pretraining in a fully self-supervised manner. To this end, we present FILS, a novel self-supervised video Feature prediction In semantic Language Space (FILS). The vision model can capture valuable structured information by correctly predicting masked feature semantics in language space. It is learned using a patch-wise video-text contrastive strategy, in which the text representations act as prototypes for transforming vision features into a language space, which are then used as targets for semantically meaningful feature prediction using our masked encoder-decoder structure. FILS demonstrates remarkable transferability on downstream action recognition tasks, achieving state-of-the-art on challenging egocentric datasets, like Epic-Kitchens, Something-SomethingV2, Charades-Ego, and EGTEA, using ViT-Base. Our efficient method requires less computation and smaller batches compared to previous works.<|reference_end|>
@article{ahmadian2024fils:,
    title={FILS: Self-Supervised Video Feature Prediction In Semantic Language
  Space},
    author={Mona Ahmadian, Frank Guerin, Andrew Gilbert},
    journal={arXiv preprint arXiv:2406.03447},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.03447},
    primaryClass={cs.CV cs.AI cs.LG}
}