benchmarks, (3) has low training and inference cost, and (4)
can be applied to clips of over one minute, thus enabling
long-term video modeling. In the future, we plan to extend our method to other video analysis tasks such as action
localization, video captioning and question-answering.
Appendix
A. Implementation Details
Our TimeSformer implementation is built using PySlowFast (Fan et al., 2020) and pytorch-image-models (Wightman, 2019) packages. Below, we describe speciﬁc implementation details regarding the training and inference procedures of our model.
Training. We train our model for 15epochs with an initial
learning rate of 0:005, which is divided by 10at epochs
11;and14. During training, we ﬁrst resize the shorter
side of the video to a random value in [256;320]. We then
randomly sample a 224224crop from the resized video.
For our high-resolution model, TimeSformer-HR, we resize
the shorter side of the video to a random value in [448;512],
and then randomly sample a 448448crop. We randomlysample clips from the full-length videos with a frame rate
of1=32. The batch size is set to 16. We train all our models
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani