lead to similar accuracy.
(i.e.,448GPU hours), its accuracy drops to 70:0%. Similarly, training I3D using a similar computational budget (i.e.,
444GPU hours) leads to a lower accuracy of 71:0%. This
highlights the fact that some of the latest 3D CNNs (Feichtenhofer et al., 2019b; Feichtenhofer, 2020) require a very
long optimization schedule to achieve good performance
(even when using ImageNet pretraining). In contrast, TimeSformer provides a more efﬁcient alternative to labs that do
not have access to hundreds of GPUs.
The Importance of Pretraining. Due to a large number
of parameters, training our model from scratch is difﬁcult.
Thus, before training TimeSformer on video data, we initialize it with weights learned from ImageNet. In contrast,
SlowFast can be learned on video data from scratch although
at the expense of a very high training cost (see Table 2). We
also attempted to train TimeSformer on Kinetics-400 directly, without any ImageNet pretraining. By using a longer
training schedule and more data augmentations, we found
it possible to train the model from scratch, albeit to a much
lower video level accuracy of 64:8%. Thus, based on these
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani