title: An Empirical Study of Autoregressive Pre-training from Videos
summary: We empirically study autoregressive pre-training from videos. To perform our
  study, we construct a series of autoregressive video models, called Toto. We
  treat videos as sequences of visual tokens and train transformer models to
  autoregressively predict future tokens. Our models are pre-trained on a diverse
  dataset of videos and images comprising over 1 trillion visual tokens. We
  explore different architectural, training, and inference design choices. We
  evaluate the learned visual representations on a range of downstream tasks
  including image recognition, video classification, object tracking, and
  robotics. Our results demonstrate that, despite minimal inductive biases,
  autoregressive pre-training leads to competitive performance across all
  benchmarks. Finally, we find that scaling our video models results in similar
  scaling curves to those seen in language models, albeit with a different rate.
  More details at https://brjathu.github.io/toto/
paper_authors._id: 67809ce9063dd44ffb1de7a7
paper_authors.name: Jathushan Rajasegaran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:22:39.142Z
paper_authors._id: 63895b3a43d8b0797a0d0406
paper_authors.avatarUrl: /avatars/04d9952dff70f85d4e481ec80ae818cb.svg
paper_authors.fullname: Jathushan Rajasegaran
paper_authors.type: user
paper_authors.user: brjathu
paper_authors._id: 67809ce9063dd44ffb1de7a8
paper_authors.name: Ilija Radosavovic
paper_authors._id: 67809ce9063dd44ffb1de7a9
paper_authors.name: Rahul Ravishankar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:23:24.331Z
paper_authors._id: 667f3f8c5595354e745a1eb8
paper_authors.avatarUrl: /avatars/7bdf766b61709e318c86795b2f48d424.svg
paper_authors.fullname: Rahul Ravishankar
paper_authors.type: user
paper_authors.user: rravishankar
paper_authors._id: 67809ce9063dd44ffb1de7aa
paper_authors.name: Yossi Gandelsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:23:33.999Z
paper_authors._id: 62f6942205ca68c0e0ff7b97
paper_authors.avatarUrl: /avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
paper_authors.fullname: Yossi Gandelsman
paper_authors.type: user
paper_authors.user: yossig
paper_authors._id: 67809ce9063dd44ffb1de7ab
paper_authors.name: Christoph Feichtenhofer
paper_authors._id: 67809ce9063dd44ffb1de7ac
paper_authors.name: Jitendra Malik
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-10T08:24:49.391Z
paper_authors._id: 65369a95605a07338de78ab0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sGFjOjLT2akN-sn5beVWL.jpeg
paper_authors.fullname: Jitendra Malik 
paper_authors.type: user
paper_authors.user: jitendra1995
paper_title: An Empirical Study of Autoregressive Pre-training from Videos
paper_summary: We empirically study autoregressive pre-training from videos. To perform our
  study, we construct a series of autoregressive video models, called Toto. We
  treat videos as sequences of visual tokens and train transformer models to
  autoregressively predict future tokens. Our models are pre-trained on a diverse
  dataset of videos and images comprising over 1 trillion visual tokens. We
  explore different architectural, training, and inference design choices. We
  evaluate the learned visual representations on a range of downstream tasks
  including image recognition, video classification, object tracking, and
  robotics. Our results demonstrate that, despite minimal inductive biases,
  autoregressive pre-training leads to competitive performance across all
  benchmarks. Finally, we find that scaling our video models results in similar
  scaling curves to those seen in language models, albeit with a different rate.
  More details at https://brjathu.github.io/toto/
paper_ai_summary: Autoregressive pre-training on a massive dataset of videos and images using transformer models yields competitive performance across various tasks and exhibits scaling behavior similar to language models.
paper_ai_keywords: autoregressive pre-training transformer models visual tokens image recognition video classification object tracking robotics scaling curves language models