strong performance at low inference cost.
methods use 10temporal clips with 3spatial crops (for a total of 30space-time views) during inference, TimeSformer
achieves solid accuracy with only 3views (3 spatial crops),
which reduces the inference cost. Our long-range variant,
TimeSformer-L achieves a top-1 accuracy of 80:7%. Furthermore, our default TimeSformer has the lowest inference
cost among recent state-of-the-art models. Yet, it still provides a solid accuracy of 78:0%, outperforming many more
costly models.
We also measured the actual inference runtime on 20Kvalidation videos of Kinetics-400 (using 8Tesla V100 GPUs).
Whereas SlowFast takes 14:88hours to complete the inference, TimeSformer, TimeSformer-HR, and TimeSformer-L
take36minutes, 1:06hours and 2:6hours, respectively.
Thus, even though SlowFast and TimeSformer-L have comparable cost in terms of TFLOPs, in practice the runtimes
of all our versions of TimeSformer are much lower.
In Table 6, we also present our results on Kinetics-600.
Just like on Kinetics-400, we observe that TimeSformer
performs well on this benchmark, outperforming all prior
methods.
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani