departure from current convolutional models, which are
typically limited to processing inputs of 832frames.
8 32 64 96Number of Input Frames65707580Clip Accuracy224 336 448 560Spatial Crop Side (Px)65707580Clip AccuracyFigure 5. Clip-level accuracy on Kinetics-400 as a function of spatial crop size in pixels (left), and the number of input frames (right).
Positional Embedding K400 SSv2
None 75.4 45.8
Space-only 77.8 52.5
Space-Time 78.0 59.5
Table 4. Ablation on positional embeddings. The version of TimeSformer using space-time positional embeddings yields the highest
accuracy on both Kinetics-400 and SSv2.
4.4. The Importance of Positional Embeddings
To investigate the importance of our learned spatiotemporal positional embeddings, we also conduct experiments
with a few variants of TimeSformer that use: (1) no positional embedding, (2) space-only positional embedding,
and (3) space-time positional embedding. We report these
results in Table 4. Based on these results, we observe
that the variant of our model that uses space-time positional embeddings produces the best accuracy on both
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani