task demonstrated in the video (one out of the 23K classes),
which can be used for supervised training. Thus, it is a good
benchmark to assess the ability of a model to recognize
activities exhibited over very long temporal extents.
For this evaluation, we consider only categories that have
at least 100 video examples. This gives a subset of
HowTo100M corresponding to 120Kvideos spanning 1059
task categories. We randomly partition this collection into
85Ktraining videos and 35Ktesting videos.
Is Space-Time Attention All You Need for Video Understanding?
Method # Input
FramesSingle Clip
Coverage# Test
ClipsTop-1
Acc
SlowFast 8 8.5s 48 48.2
SlowFast 32 34.1s 12 50.8
SlowFast 64 68.3s 6 51.5
SlowFast 96 102.4s 4 51.2
TimeSformer 8 8.5s 48 56.8
TimeSformer 32 34.1s 12 61.2
TimeSformer 64 68.3s 6 62.2
TimeSformer 96 102.4s 4 62.6
Table 8. Long-term task classiﬁcation on HowTo100M. Given a
video spanning several minutes, the goal is to predict the long-term
task demonstrated in the video (e.g., cooking breakfast, cleaning
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani