title: Test-Time Training on Video Streams
summary: Prior work has established test-time training (TTT) as a general framework to
  further improve a trained model at test time. Before making a prediction on
  each test instance, the model is trained on the same instance using a
  self-supervised task, such as image reconstruction with masked autoencoders. We
  extend TTT to the streaming setting, where multiple test instances - video
  frames in our case - arrive in temporal order. Our extension is online TTT: The
  current model is initialized from the previous model, then trained on the
  current frame and a small window of frames immediately before. Online TTT
  significantly outperforms the fixed-model baseline for four tasks, on three
  real-world datasets. The relative improvement is 45% and 66% for instance and
  panoptic segmentation. Surprisingly, online TTT also outperforms its offline
  variant that accesses more information, training on all frames from the entire
  test video regardless of temporal order. This differs from previous findings
  using synthetic videos. We conceptualize locality as the advantage of online
  over offline TTT. We analyze the role of locality with ablations and a theory
  based on bias-variance trade-off.
paper_authors._id: 64ae18a680f308a395f6cfa8
paper_authors.name: Renhao Wang
paper_authors._id: 64ae18a680f308a395f6cfa9
paper_authors.name: Yu Sun
paper_authors._id: 64ae18a680f308a395f6cfaa
paper_authors.name: Yossi Gandelsman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:10:33.540Z
paper_authors._id: 62f6942205ca68c0e0ff7b97
paper_authors.avatarUrl: /avatars/b0600de531b4c88f4aa7c30c5ceb06e1.svg
paper_authors.fullname: Yossi Gandelsman
paper_authors.type: user
paper_authors.user: yossig
paper_authors._id: 64ae18a680f308a395f6cfab
paper_authors.name: Xinlei Chen
paper_authors._id: 64ae18a680f308a395f6cfac
paper_authors.name: Alexei A. Efros
paper_authors._id: 64ae18a680f308a395f6cfad
paper_authors.name: Xiaolong Wang
paper_title: Test-Time Training on Video Streams
paper_summary: Prior work has established test-time training (TTT) as a general framework to
  further improve a trained model at test time. Before making a prediction on
  each test instance, the model is trained on the same instance using a
  self-supervised task, such as image reconstruction with masked autoencoders. We
  extend TTT to the streaming setting, where multiple test instances - video
  frames in our case - arrive in temporal order. Our extension is online TTT: The
  current model is initialized from the previous model, then trained on the
  current frame and a small window of frames immediately before. Online TTT
  significantly outperforms the fixed-model baseline for four tasks, on three
  real-world datasets. The relative improvement is 45% and 66% for instance and
  panoptic segmentation. Surprisingly, online TTT also outperforms its offline
  variant that accesses more information, training on all frames from the entire
  test video regardless of temporal order. This differs from previous findings
  using synthetic videos. We conceptualize locality as the advantage of online
  over offline TTT. We analyze the role of locality with ablations and a theory
  based on bias-variance trade-off.
paper_ai_summary: Online test-time training (TTT) framework for video frames improves performance by using a small temporal window compared to fixed-model or offline TTT methods.
paper_ai_keywords: test-time training TTT self-supervised task masked autoencoders streaming setting video frames online TTT instance segmentation panoptic segmentation bias-variance trade-off