PooDLe: Pooled and dense self-supervised learning from naturalistic videos
<|reference_start|>PooDLe: Pooled and dense self-supervised learning from naturalistic videos: Self-supervised learning has driven significant progress in learning from single-subject, iconic images. However, there are still unanswered questions about the use of minimally-curated, naturalistic video data, which contain dense scenes with many independent objects, imbalanced class distributions, and varying object sizes. In this paper, we propose a novel approach that combines an invariance-based SSL objective on pooled representations with a dense SSL objective that enforces equivariance to optical flow warping. Our findings indicate that a unified objective applied at multiple feature scales is essential for learning effective image representations from high-resolution, naturalistic videos. We validate our approach on the BDD100K driving video dataset and the Walking Tours first-person video dataset, demonstrating its ability to capture spatial understanding from a dense objective and semantic understanding via a pooled representation objective.<|reference_end|>
@article{wang2024poodle:,
    title={PooDLe: Pooled and dense self-supervised learning from naturalistic
  videos},
    author={Alex N. Wang, Christopher Hoang, Yuwen Xiong, Yann LeCun, Mengye Ren},
    journal={arXiv preprint arXiv:2408.11208},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.11208},
    primaryClass={cs.CV cs.LG}
}