training videos. All models are pretrained on ImageNet-1K.
The Impact of Video-Data Scale. To understand the
effects of video-data scale on performance, we trained
TimeSformer on different subsets of K400 and SSv2:
f25%;50%;75%;100%gof the full datasets. We show
these results in Figure 4, where we also compare our method
with SlowFast R50 (Feichtenhofer et al., 2019b), and I3D
R50 (Carreira & Zisserman, 2017) trained on the same subsets and using the same pretraining. Since we do not have
access to a ResNet pretrained on ImageNet-21K, we use
ImageNet-1K pretraining for all 3architectures.
The results of Figure 4 show that, on K400, TimeSformer
outperforms the other models for all training subsets. However, we observe a different trend on SSv2, where TimeSformer is the strongest model only when trained on 75% or
100% of the full data. This may be explained by the fact that
compared to K400, SSv2 requires learning more complex
temporal patterns, and thus more examples are needed by
TimeSformer to learn effectively those patterns.
4.3. Varying the Number of Tokens
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani