from the image space to the space-time 3D volume. Our
proposed model, named “TimeSformer” (from Time-Space
Transformer), views the video as a sequence of patches extracted from the individual frames. As in ViT, each patch is
linearly mapped into an embedding and augmented with positional information. This makes it possible to interpret the
resulting sequence of vectors as token embeddings which
can be fed to a Transformer encoder, analogously to the
token features computed from words in NLP.
One downside of self-attention in standard Transformer is
that it requires computing a similarity measure for all pairs
of tokens. In our setting, this is computationally costly due
to the large number of patches in the video. To address these
challenges, we propose several scalable self-attention designs over the space-time volume and empirically evaluate
them over large-scale action classiﬁcation datasets. Among
the proposed schemes, we found that the best design is represented by a “divided attention” architecture which separately
applies temporal attention and spatial attention within each
block of the network. Compared to the established paradigm
of convolution-based video architecture, TimeSformer follows a radically different design. Yet, it achieves accuracy
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani