to address various video-language tasks, such as captioning (Zhou et al., 2018), question-answering (Yang et al.,
2020) and dialog (Le et al., 2019). Finally, multimodal
video-text transformers (Sun et al., 2019; Li et al., 2020a)
have also been trained or pretrained in unsupervised fashion
by adopting masked-token pretext tasks adapted from the
language domain (Devlin et al., 2018; Radford et al., 2018).
3. The TimeSformer Model
Input clip. The TimeSformer takes as input a clip X2
RHW3Fconsisting of FRGB frames of size HW
sampled from the original video.
Decomposition into patches. Following the ViT (Dosovitskiy et al., 2020), we decompose each frame into N
non-overlapping patches, each of size PP, such that
theNpatches span the entire frame, i.e., N=HW=P2.
We ﬂatten these patches into vectors x(p;t)2R3P2with
Is Space-Time Attention All You Need for Video Understanding?
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani