title: Learning Video Representations without Natural Videos
summary: In this paper, we show that useful video representations can be learned from
  synthetic videos and natural images, without incorporating natural videos in
  the training. We propose a progression of video datasets synthesized by simple
  generative processes, that model a growing set of natural video properties
  (e.g. motion, acceleration, and shape transformations). The downstream
  performance of video models pre-trained on these generated datasets gradually
  increases with the dataset progression. A VideoMAE model pre-trained on our
  synthetic videos closes 97.2% of the performance gap on UCF101 action
  classification between training from scratch and self-supervised pre-training
  from natural videos, and outperforms the pre-trained model on HMDB51.
  Introducing crops of static images to the pre-training stage results in similar
  performance to UCF101 pre-training and outperforms the UCF101 pre-trained model
  on 11 out of 14 out-of-distribution datasets of UCF101-P. Analyzing the
  low-level properties of the datasets, we identify correlations between frame
  diversity, frame similarity to natural data, and downstream performance. Our
  approach provides a more controllable and transparent alternative to video data
  curation processes for pre-training.
paper_authors._id: 6724357b3a6caef9c87e1633
paper_authors.name: Xueyang Yu
paper_authors._id: 6724357b3a6caef9c87e1634
paper_authors.name: Xinlei Chen
paper_authors._id: 6724357b3a6caef9c87e1635
paper_authors.name: Yossi Gandelsman
paper_title: Learning Video Representations without Natural Videos
paper_summary: In this paper, we show that useful video representations can be learned from
  synthetic videos and natural images, without incorporating natural videos in
  the training. We propose a progression of video datasets synthesized by simple
  generative processes, that model a growing set of natural video properties
  (e.g. motion, acceleration, and shape transformations). The downstream
  performance of video models pre-trained on these generated datasets gradually
  increases with the dataset progression. A VideoMAE model pre-trained on our
  synthetic videos closes 97.2% of the performance gap on UCF101 action
  classification between training from scratch and self-supervised pre-training
  from natural videos, and outperforms the pre-trained model on HMDB51.
  Introducing crops of static images to the pre-training stage results in similar
  performance to UCF101 pre-training and outperforms the UCF101 pre-trained model
  on 11 out of 14 out-of-distribution datasets of UCF101-P. Analyzing the
  low-level properties of the datasets, we identify correlations between frame
  diversity, frame similarity to natural data, and downstream performance. Our
  approach provides a more controllable and transparent alternative to video data
  curation processes for pre-training.
paper_ai_summary: A synthetic video dataset enables effective pre-training of VideoMAE models for video tasks without using natural videos, achieving high performance on various classification and out-of-distribution datasets.
paper_ai_keywords: video representations synthetic videos natural images generative processes motion acceleration shape transformations downstream performance video models VideoMAE UCF101 HMDB51 frame diversity frame similarity out-of-distribution datasets UCF101-P