of the pretraining dataset that we use.
4.7. Additional Ablations
Smaller & Larger Transformers. In addition to the “Base”
ViT model (Dosovitskiy et al., 2020), we also experimented
with the “Large” ViT. We report that this yielded results 1%
Figure 7. Visualization of space-time attention from the output
token to the input space on Something-Something-V2. Our model
learns to focus on the relevant parts in the video in order to perform
spatiotemporal reasoning.
ViT-50 -40 -30 -20 -10 0 10 20 30 40 50-50-40-30-20-1001020304050
-60 -40 -20 0 20 40-40-30-20-1001020304050       TimeSformer w/ Divided            Space-Time Attention-60 -40 -20 0 20 40 60-60-40-200204060    TimeSformer  w/ Space Attention
Figure 8. Feature visualization with t-SNE (van der Maaten & Hinton, 2008) on Something-Something-V2. Each video is visualized
as a point. Videos belonging to the same action category have the
same color. The TimeSformer with divided space-time attention
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani