model of Eq. 5, Divided Attention performs only (N+F+2)
comparisons per patch. Our experiments demonstrate that
this space-time factorization is not only more efﬁcient but it
also leads to improved classiﬁcation accuracy.
We have also experimented with a “Sparse Local Global”
(L+G) and an “Axial” (T+W+H) attention models. Their
architectures are illustrated in Fig. 1, while Fig. 2 shows
the patches considered for attention by these models. For
each patch (p;t), (L+G) ﬁrst computes a local attention by
considering the neighboring FH=2W=2patches and
then calculates a sparse global attention over the entire clip
using a stride of 2 patches along the temporal dimension and
also the two spatial dimensions. Thus, it can be viewed as a
faster approximation of full spatiotemporal attention using a
local-global decomposition and a sparsity pattern, similar to
that used in (Child et al., 2019). Finally, “Axial” attention
decomposes the attention computation in three distinct steps:
over time, width and height. A decomposed attention over
the two spatial axes of the image was proposed in (Ho
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani