of shorter-range information. Conversely, the self-attention
mechanism can be applied to capture both local as well as
global long-range dependencies by directly comparing feature activations at all space-time locations, much beyond the
receptive ﬁeld of traditional convolutional ﬁlters. Finally,arXiv:2102.05095v4  [cs.CV]  9 Jun 2021
Is Space-Time Attention All You Need for Video Understanding?
despite the advances in GPU hardware acceleration, training
deep CNNs remains very costly, especially when applied to
high-resolution and long videos. Recent work in the stillimage domain (Dosovitskiy et al., 2020; Carion et al., 2020;
Zhao et al., 2020) has demonstrated that Transformers enjoy
faster training and inference compared to CNNs, making it
possible to construct models with larger learning capacity
for the same computational budget.
Motivated by these observations, we propose a video architecture built exclusively on self-attention. We adapt the
image model “Vision Transformer” (ViT) (Dosovitskiy et al.,
2020) to video by extending the self-attention mechanism
from the image space to the space-time 3D volume. Our
proposed model, named “TimeSformer” (from Time-Space
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani