mainly from decomposing the video into a sequence of
frame-level patches and then feeding linear embeddings of
these patches as input token embeddings to a Transformer.
This strategy was recently introduced in Vision Transformers (ViT) (Dosovitskiy et al., 2020) which were shown to
deliver impressive performance on image categorization. In
this work, we build on the ViT design, and extend it to video
by proposing and empirically comparing several scalable
schemes for space-time self-attention over videos.
While Transformers have been recently used for video generation (Weissenborn et al., 2020), we are not aware of prior
video recognition architectures using self-attention as the
exclusive building block. However, we note that Transformers have been adopted on top of convolutional feature
maps for action localization and recognition (Girdhar et al.,
2019), video classiﬁcation (Wang et al., 2018b; Chen et al.,
2018b), and group activity recognition (Gavrilyuk et al.,
2020). We also note that there is a wide literature based on
the use of text Transformers combined with video CNNs
to address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al.,
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani