ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning
<|reference_start|>ARVideo: Autoregressive Pretraining for Self-Supervised Video Representation Learning: This paper presents a new self-supervised video representation learning framework, ARVideo, which autoregressively predicts the next video token in a tailored sequence order. Two key designs are included. First, we organize autoregressive video tokens into clusters that span both spatially and temporally, thereby enabling a richer aggregation of contextual information compared to the standard spatial-only or temporal-only clusters. Second, we adopt a randomized spatiotemporal prediction order to facilitate learning from multi-dimensional data, addressing the limitations of a handcrafted spatial-first or temporal-first sequence order. Extensive experiments establish ARVideo as an effective paradigm for self-supervised video representation learning. For example, when trained with the ViT-B backbone, ARVideo competitively attains 81.2% on Kinetics-400 and 70.9% on Something-Something V2, which are on par with the strong benchmark set by VideoMAE. Importantly, ARVideo also demonstrates higher training efficiency, i.e., it trains 14% faster and requires 58% less GPU memory compared to VideoMAE.<|reference_end|>
@article{ren2024arvideo:,
    title={ARVideo: Autoregressive Pretraining for Self-Supervised Video
  Representation Learning},
    author={Sucheng Ren, Hongru Zhu, Chen Wei, Yijiang Li, Alan Yuille, Cihang Xie},
    journal={arXiv preprint arXiv:2405.15160},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.15160},
    primaryClass={cs.CV}
}