Beyond image categorization, Relation Networks (Hu et al.,
2018) and DETR (Carion et al., 2020) use self-attention on
top of convolutional feature maps for object detection.Our method is more closely related to image networks leveraging self-attention as a substitute for convolution (Parmar
et al., 2018; Ramachandran et al., 2019; Cordonnier et al.,
2020; Zhao et al., 2020). Since these works use individual
pixels as queries, in order to maintain a manageable computational cost and a small memory consumption, they must
restrict the scope of self-attention to local neighborhoods or
use global self-attention on heavily downsized versions of
the image. Alternative strategies for scalability to full images include sparse key-value sampling (Child et al., 2019)
or constraining the self-attention to be calculated along the
spatial axes (Ho et al., 2019; Huang et al., 2019; Wang
et al., 2020b). A few of the self-attention operators considered in our experiments adopt similar sparse and axial
computation, although generalized to the spatiotemporal
volume. However, the efﬁciency of our approach stems
mainly from decomposing the video into a sequence of
frame-level patches and then feeding linear embeddings of
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani