For this ﬁrst set of experiments we start from a ViT pretrained on ImageNet-21K. In Table 1, we present the results
obtained with TimeSformer for the ﬁve proposed space-time
attention schemes on Kinetics-400 (K400) and SomethingSomething-V2 (SSv2). First, we note that TimeSformer
with space-only attention (S) performs well on K400. This
is an interesting ﬁnding. Indeed, prior work (Sevilla-Lara
et al., 2021) has shown that on K400, spatial cues are more
important than temporal information in order to achieve
strong accuracy. Here, we show that it is possible to obtain
solid accuracy on K400 without any temporal modeling.
Note, however, that space-only attention performs poorly on
SSv2. This stresses the importance of temporal modeling
on this latter dataset.
Furthermore, we observe that divided space-time attention
achieves the best accuracy on both K400 and SSv2. This
makes sense because compared to joint space-time attention,
divided space-time attention has a larger learning capacity
(see Table 1) as it contains distinct learning parameters for
temporal attention and spatial attention.
In Figure 3, we also compare the computational cost of joint
space-time versus divided space-time attention when using
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani