seconds, we would sample 48test clips to cover a video of
410seconds. Video-level classiﬁcation is done by averaging
the clip predictions.
From the results in Table 8 we ﬁrst note that, for the same
single clip coverage, TimeSformer outperforms the corresponding SlowFast by a large margin of 811%. We also
observe that longer-range TimeSformers do better, i.e., our
longest-range variant achieves the best video-level classiﬁcation accuracy. These results suggest that our model is highly
suitable for tasks that require long-term video modeling.
We also experimented with ﬁnetuning TimeSformer directly
from a ViT pretrained on ImageNet-1K and ImageNet21K (skipping the Kinetics-400 training). We report that
when pretrained only on ImageNet-1K, our model achieves
top-1 accuracies of 52:8;58:4;59:2;59:4for8;32;64;96
frame inputs, respectively. When considering ImagNet21K pretraining, TimeSformer produces top-1 accuracies of
56:0;59:2;60:2;62:1for8;32;64;96frame inputs, respectively. These results demonstrate that our model can effectively exploit long-range temporal dependencies regardless
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani