block of the network. Compared to the established paradigm
of convolution-based video architecture, TimeSformer follows a radically different design. Yet, it achieves accuracy
comparable, and in some cases superior, to the state-of-theart in this ﬁeld. We also show that our model can be used
for long-range modeling of videos spanning many minutes.
2. Related Work
Our approach is inﬂuenced by recent works that use selfattention for image classiﬁcation, either in combination with
the convolution operator or even as a full replacement for it.
Within the former class, Non-Local Networks (Wang et al.,
2018b) employ a non-local mean that effectively generalizes
the self-attention function of Transformers (Vaswani et al.,
2017b). Bello et al. (Bello et al., 2019) propose a 2D selfattention mechanism that is competitive as a replacement of
2D convolution but gives even stronger results when used to
augment convolutional features with self-attention features.
Beyond image categorization, Relation Networks (Hu et al.,
2018) and DETR (Carion et al., 2020) use self-attention on
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani