of1=32. The batch size is set to 16. We train all our models
using synchronized SGD across 32GPUs. The momentum
is set to 0:9, while the weight decay is set to 0:0001 .
Unless otherwise noted, in our experiments we use the
“Base” ViT model (Dosovitskiy et al., 2020). Temporal and
spatial attention layers in each block are initialized with the
same weights, which are obtained from the corresponding
attention layer in ViT.
Inference. As discussed in the main draft, during inference
we sample a single temporal clip in the middle of the video.
We scale the shorter spatial side of a video to 224pixels (or
448for TimeSformer-HR) and take 3crops of size 224224
(448448for TimeSformer-HR) to cover a larger spatial
extent within the clip. The ﬁnal prediction is obtained by
averaging the softmax scores of these 3predictions.
Other models in our comparison. To train I3D (Carreira
& Zisserman, 2017), and SlowFast (Feichtenhofer et al.,
2019b), we use the training protocols that were used in the
original papers. For I3D, we initialize it with a 2D ImageNet
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani