title: An Empirical Study of Autoregressive Pre-training from Videos
summary: We empirically study autoregressive pre-training from videos. To perform our
  study, we construct a series of autoregressive video models, called Toto. We
  treat videos as sequences of visual tokens and train transformer models to
  autoregressively predict future tokens. Our models are pre-trained on a diverse
  dataset of videos and images comprising over 1 trillion visual tokens. We
  explore different architectural, training, and inference design choices. We
  evaluate the learned visual representations on a range of downstream tasks
  including image recognition, video classification, object tracking, and
  robotics. Our results demonstrate that, despite minimal inductive biases,
  autoregressive pre-training leads to competitive performance across all
  benchmarks. Finally, we find that scaling our video models results in similar
  scaling curves to those seen in language models, albeit with a different rate.
  More details at https://brjathu.github.io/toto/
paper_authors._id: 67809ce9063dd44ffb1de7a7
paper_authors.name: Jathushan Rajasegaran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:22:39.142Z
paper_authors._id: 63895b3a43d8b0797a0d0406
paper_authors.avatarUrl: /avatars/04d9952dff70f85d4e481ec80ae818cb.svg
paper_authors.fullname: Jathushan Rajasegaran
paper_authors.type: user
paper_authors.user: brjathu
paper_authors._id: 67809ce9063dd44ffb1de7a8
paper_authors.name: Ilija Radosavovic
paper_authors._id: 67809ce9063dd44ffb1de7a9
paper_authors.name: Rahul Ravishankar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:23:24.331Z
paper_authors._id: 667f3f8c5595354e745a1eb8
paper_authors.avatarUrl: /avatars/7bdf766b61709e318c86795b2f48d424.svg
paper_authors.fullname: Rahul Ravishankar
paper_authors.type: user
paper_authors.user: rravishankar
paper_authors._id: 67809ce9063dd44ffb1de7aa
paper_authors.name: Yossi Gandelsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:23:33.999Z
paper_authors._id: 62f6942205ca68c0e0ff7b97
paper_authors.avatarUrl: /avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
paper_authors.fullname: Yossi Gandelsman
paper_authors.type: user
paper_authors.user: yossig
paper_authors._id: 67809ce9063dd44ffb1de7ab
paper_authors.name: Christoph Feichtenhofer
paper_authors._id: 67809ce9063dd44ffb1de7ac
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:24:49.391Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_title: An Empirical Study of Autoregressive Pre-training from Videos
paper_summary: We empirically study autoregressive pre-training from videos. To perform our
  study, we construct a series of autoregressive video models, called Toto. We
  treat videos as sequences of visual tokens and train transformer models to
  autoregressively predict future tokens. Our models are pre-trained on a diverse
  dataset of videos and images comprising over 1 trillion visual tokens. We
  explore different architectural, training, and inference design choices. We
  evaluate the learned visual representations on a range of downstream tasks
  including image recognition, video classification, object tracking, and
  robotics. Our results demonstrate that, despite minimal inductive biases,
  autoregressive pre-training leads to competitive performance across all
  benchmarks. Finally, we find that scaling our video models results in similar
  scaling curves to those seen in language models, albeit with a different rate.
  More details at https://brjathu.github.io/toto/
paper_ai_keywords: autoregressive pre-training autoregressive video models visual tokens transformer models autoregressive prediction diverse dataset architectural, training, and inference design choices visual representations image recognition video classification object tracking robotics competitive performance benchmarking scaling video models language models