clips (over one minute long). Code and models are available at: https://github.com/
facebookresearch/TimeSformer .
1. Introduction
Over the last few years, the ﬁeld of natural language processing (NLP) has been revolutionized by the emergence of
methods based on self-attention (Vaswani et al., 2017a). Because of their excellent capabilities at capturing long-range
dependencies among words as well as their training scalability, self-attention architectures, such as the Transformer
model, represent the current state-of-the-art across a wide
range of language tasks, including machine translation (Ott
et al., 2018; Chen et al., 2018a), question answering (Devlin et al., 2019; Dai et al., 2019), and autoregressive word
generation (Radford et al., 2019; Brown et al., 2020).
Video understanding shares several high-level similarities
1Facebook AI2Dartmouth College. Correspondence to: Gedas
Bertasius <gberta@seas.upenn.edu>.with NLP. First of all, videos and sentences are both sequential. Furthermore, precisely as the meaning of a word can
often be understood only by relating it to the other words in
the sentence, it may be argued that atomic actions in shortterm segments need to be contextualized with the rest of the
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani