video spanning several minutes, the goal is to predict the long-term
task demonstrated in the video (e.g., cooking breakfast, cleaning
house, etc). We evaluate a few variants of SlowFast and TimeSformer on this task. “Single Clip Coverage” denotes the number
of seconds spanned by a single clip. “# Test Clips” is the average
number of clips needed to cover the entire video during inference.
All models in this comparison are pretrained on Kinetics-400.
We present our results in Table 8. As our baselines, we use
four variants of SlowFast R101, all operating on video clips
sampled at a frame rate of 1=32but having varying number
of frames: 8;32;64and96. We use the same four conﬁgurations for TimeSformer, starting from a ViT pretrained
on ImageNet-21K. All models in this comparison are pretrained on Kinetics-400 before ﬁnetuning on HowTo100M.
During inference, for each method, we sample as many
non-overlapping temporal clips as needed to cover the full
temporal extent of a video, e.g., if a single clip spans 8:5
seconds, we would sample 48test clips to cover a video of
410seconds. Video-level classiﬁcation is done by averaging
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani