training schedule and more data augmentations, we found
it possible to train the model from scratch, albeit to a much
lower video level accuracy of 64:8%. Thus, based on these
results, for all subsequent studies we continued to use ImageNet for pretraining (Deng et al., 2009)
In Table 3 we study the beneﬁts of ImageNet-1K vs
ImageNet-21K pretraining on K400 and SSv2. For these
experiments, we use three variants of our model: (1) TimeSformer , which is the default version of our model operating
on8224224video clips, (2) TimeSformer-HR , a high
spatial resolution variant that operates on 16448448
video clips, and lastly (3) TimeSformer-L , a long-range
conﬁguration of our model that operates on 96224224
video clips with frames sampled at a rate of 1=4.
Based on the results in Table 3, we observe that ImageNet21K pretraining is beneﬁcial for K400, where it leads to
a consistently higher accuracy compared to ImageNet-1K
pretraining. On the other hand, on SSv2, we observe that
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani