inference cost ( 1:11TFLOPs) despite containing fewer parameters ( 28:0M). This suggests that TimeSformer is better
suited for settings that involve large-scale learning. In contrast, the large computational cost of modern 3D CNNs
makes it difﬁcult to further increase their model capacity
while also maintaining efﬁciency.
Video Training Time. One signiﬁcant advantage of ImageNet pretraining is that it enables very efﬁcient training of
TimeSformer on video data. Conversely, state-of-the-art 3D
CNNs are much more expensive to train even if pretrained
on image datasets. In Table 2, we compare the video training time on Kinetics-400 (in Tesla V100 GPU hours) of
TimeSformer to that of SlowFast and I3D. Starting from a
ResNet50 pretrained on ImageNet-1K, SlowFast 88R50
requires 3 840 Tesla V100 GPU hours in order to reach an
accuracy of 75:6%on Kinetics-400. Training I3D, under
similar settings, requires 1 440 Tesla V100 GPU hours for a
73:4%accuracy. In contrast, TimeSformer, also pretrained
on ImageNet-1K, only requires 416Tesla V100 GPU hours
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani