a consistently higher accuracy compared to ImageNet-1K
pretraining. On the other hand, on SSv2, we observe that
ImageNet-1K and ImageNet-21K pretrainings lead to similar accuracy. This makes sense as SSv2 requires complex
spatiotemporal reasoning, whereas K400 is biased more towards spatial scene information, and thus, it beneﬁts more
from the features learned on the larger pretraining dataset.
Is Space-Time Attention All You Need for Video Understanding?
60K 120K 180K 240K# of Training Videos60657075AccuracyKinetics
TimeSformerSlowFastI3D42K 85K 127K 170K# of Training Videos354045505560AccuracySomething-Something-V2
TimeSformerSlowFastI3D
Figure 4. Accuracy on Kinetics-400 (K400), and SomethingSomething-V2 (SSv2) as a function of the number of training
videos. On K400, TimeSformer performs best in all cases. On
SSv2, which requires more complex temporal reasoning, TimeSformer outperforms the other models only when using enough
training videos. All models are pretrained on ImageNet-1K.
The Impact of Video-Data Scale. To understand the
effects of video-data scale on performance, we trained
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani