often be understood only by relating it to the other words in
the sentence, it may be argued that atomic actions in shortterm segments need to be contextualized with the rest of the
video in order to be fully disambiguated. Thus, one would
expect the long-range self-attention models from NLP to be
highly effective for video modeling as well. However, in the
video domain, 2D or 3D convolutions still represent the core
operators for spatiotemporal feature learning across different video tasks (Feichtenhofer et al., 2019a; Teed & Deng,
2020; Bertasius & Torresani, 2020). While self-attention
has shown beneﬁts when applied on top of convolutional
layers (Wang et al., 2018a), to the best of our knowledge, no
attempt to use self-attention as the exclusive building block
for video recognition models has been reported.
In this work we pose the question of whether it may be
possible to build a performant convolution-free video architecture by replacing altogether the convolution operator with
self-attention. We argue that such a design has the potential to overcome a few inherent limitations of convolutional
models for video analysis. First, while their strong inductive
biases (e.g., local connectivity and translation equivariance)
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani