we investigate whether reversing the order of time-space
attention (i.e., applying spatial attention ﬁrst, then temporal) has an impact on our results. We report that applying spatial attention ﬁrst, followed by temporal attention
leads to a 0:5%drop in accuracy on both Kinetics-400, and
Something-Something-V2. We also tried a parallel spacetime self-attention. We report that it produces 0:4%lower
accuracy compared to our adopted “Divided Space-Time
Attention” scheme.
Is Space-Time Attention All You Need for Video Understanding?
4.8. Qualitative Results
Visualizing Learned Space-Time Attention. In Figure 7,
we present space-time attention visualizations obtained by
applying TimeSformer on Something-Something-V2 videos.
To visualize the learned attention, we use the Attention
Rollout scheme presented in (Abnar & Zuidema, 2020).
Our results suggest that TimeSformer learns to attend to the
relevant regions in the video in order to perform complex
spatiotemporal reasoning. For example, we can observe that
the model focuses on the conﬁguration of the hand when
visible and the object-only when not visible.
Visualizing Learned Feature Embeddings. In Figure 8,
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani