2019b), we use the training protocols that were used in the
original papers. For I3D, we initialize it with a 2D ImageNet
CNN, and then train it for 118epochs with a base learning
rate of 0:01, which is divided by 10at epochs 44and88.
We use synchronized SGD across 32GPUs following the
linear scaling recipe of Goyal et al. (2017a). We set the
momentum to 0:9, and weight decay to 0:0001 . The batch
size is set to 64. For the SlowFast model, when initialized
from ImageNet weights, we use this same exact training
protocol. When training SlowFast from scratch, we use the
training protocol described by the authors (Feichtenhofer
et al., 2019b). More speciﬁcally, in that case, the training
is done for 196epochs with a cosine learning rate schedule,
and the initial learning rate is set to 0:1. We use a linear
warm-up for the ﬁrst 34epochs starting with a learning rate
of0:01. A dropout of 0:5is used before the ﬁnal classiﬁcation layer. The momentum is set to 0:9, the weight decay is
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani