pixels. During inference, unless otherwise noted, we sample a single temporal clip in the middle of the video. We
use3spatial crops (top-left, center, bottom-right) from the
temporal clip and obtain the ﬁnal prediction by averaging
the scores for these 3crops.
8 32 64 96# of Input frames0510TFLOPsJoint Space-TimeDivided Space-Time224 336 448 560Spatial Crop (Px)0123TFLOPsJoint Space-TimeDivided Space-Time
Out of memory
Out of memoryFigure 3. We compare the video classiﬁcation cost (in TFLOPs) of
Joint Space-Time versus Divided Space-Time attention. We plot
the number of TFLOPs as a function of spatial crop size in pixels
(left), and the number of input frames (right). As we increase the
spatial resolution (left), or the video length (right), our proposed divided space-time attention leads to dramatic computational savings
compared to the scheme of joint space-time attention.
4.1. Analysis of Self-Attention Schemes
For this ﬁrst set of experiments we start from a ViT pretrained on ImageNet-21K. In Table 1, we present the results
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani