Acc.Inference
TFLOPsParams
I3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M
I3D 8x8 R50 ImageNet-1K 1440 73.4 1.11 28.0M
SlowFast R50 ImageNet-1K 448 70.0 1.97 34.6M
SlowFast R50 ImageNet-1K 3840 75.6 1.97 34.6M
SlowFast R50 N/A 6336 76.4 1.97 34.6M
TimeSformer ImageNet-1K 416 75.8 0.59 121.4M
TimeSformer ImageNet-21K 416 78.0 0.59 121.4M
Table 2. Comparing TimeSformer to SlowFast and I3D. We observe that TimeSformer has lower inference cost despite having
a larger number of parameters. Furthermore, the cost of training
TimeSformer on video data is much lower compared to SlowFast
and I3D, even when all models are pretrained on ImageNet-1K.
4.2. Comparison to 3D CNNs
In this subsection we perform an empirical study aimed
at understanding the distinguishing properties of TimeSformer compared to 3D convolutional architectures, which
have been the prominent approach to video understanding
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani