Figure 2. Visualization of the ﬁve space-time self-attention schemes studied in this work. Each video clip is viewed as a sequence of
frame-level patches with a size of 1616pixels. For illustration, we denote in blue the query patch and show in non-blue colors its
self-attention space-time neighborhood under each scheme. Patches without color are not used for the self-attention computation of the
blue patch. Multiple colors within a scheme denote attentions separately applied along different dimensions (e.g., space and time for
(T+S)) or over different neighborhoods (e.g., for (L+G)). Note that self-attention is computed for every single patch in the video clip, i.e.,
every patch serves as a query. We also note that although the attention pattern is shown for only two adjacent frames, it extends in the
same fashion to all frames of the clip.
s(`;a)
(p;t)=(`;a)
(p;t);(0;0)v(`;a)
(0;0)+NX
p0=1FX
t0=1(`;a)
(p;t);(p0;t0)v(`;a)
(p0;t0):
(7)
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani