the model focuses on the conﬁguration of the hand when
visible and the object-only when not visible.
Visualizing Learned Feature Embeddings. In Figure 8,
we also visualize the features learned by TimeSformer on
Something-Something-V2. The visualization is done using
t-SNE (van der Maaten & Hinton, 2008) where each point
represents a single video, and different colors depict different action categories. Based on this illustration, we observe
that TimeSformer with divided space-time attention learns
semantically more separable features than the TimeSformer
with space-only attention or ViT (Dosovitskiy et al., 2020).
5. Conclusion
In this work, we introduced TimeSformer, a fundamentally
different approach to video modeling compared to the established paradigm of convolution-based video networks.
We showed that it is possible to design an effective, and
scalable video architecture built exclusively on space-time
self-attention. Our method (1) is conceptually simple, (2)
achieves state-of-the-art results on major action recognition
benchmarks, (3) has low training and inference cost, and (4)
can be applied to clips of over one minute, thus enabling
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani