In this subsection we perform an empirical study aimed
at understanding the distinguishing properties of TimeSformer compared to 3D convolutional architectures, which
have been the prominent approach to video understanding
in recent years. We focus our comparison on two 3D CNN
models: 1) SlowFast (Feichtenhofer et al., 2019b), which is
the state-of-the-art in video classiﬁcation, and 2) I3D (Carreira & Zisserman, 2017), which has been shown to beneﬁt
from image-based pretraining, similarly to our own model.
We present quantitative comparisons to these two networks
in Table 2 and highlight key observations below.
Model Capacity. From Table 2, we ﬁrst observe that although TimeSformer has a large learning capacity (the number of parameters is 121:4M), it has low inference cost
(0:59in TFLOPs). In contrast, SlowFast 8x8 R50 has a
larger inference cost ( 1:97TFLOPs) despite containing only
34:6Mparameters. Similarly, I3D 8x8 R50 also has a larger
inference cost ( 1:11TFLOPs) despite containing fewer parameters ( 28:0M). This suggests that TimeSformer is better
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani