models for video analysis. First, while their strong inductive
biases (e.g., local connectivity and translation equivariance)
are undoubtedly beneﬁcial on small training sets, they may
excessively limit the expressivity of the model in settings
where there is ample availability of data and “all” can be
learned from examples. Compared to CNNs, Transformers
impose less restrictive inductive biases. This broadens the
family of functions they can represent (Cordonnier et al.,
2020; Zhao et al., 2020), and renders them better suited
to modern big-data regimes where there is less need for
strong inductive priors. Second, while convolutional kernels
are speciﬁcally designed to capture short-range spatiotemporal information, they cannot model dependencies that
extend beyond the receptive ﬁeld. While deep stacks of
convolutions (Simonyan & Zisserman, 2015; Szegedy et al.,
2015; Carreira & Zisserman, 2017) naturally extend the
receptive ﬁeld, these strategies are inherently limited in capturing long-range dependencies by means of aggregation
of shorter-range information. Conversely, the self-attention
mechanism can be applied to capture both local as well as
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani