as a point. Videos belonging to the same action category have the
same color. The TimeSformer with divided space-time attention
learns semantically more separable features than the TimeSformer
with space-only attention or ViT (Dosovitskiy et al., 2020).
worse on both Kinetics-400, and Something-Something-V2.
Given that our “Base” model already has 121Mparameters,
we suspect that the current datasets are not big enough to
justify a further increase in model capacity. We also tried
the “Small” ViT variant, which produced accuracies about
5%worse than our default “Base” ViT model.
Larger Patch Size. We also experimented with a different
patch size, i.e., P= 32 . We report that this variant of our
model produced results about 3%worse than our default
variant using P= 16 . We conjecture that the performance
decrease with P= 32 is due to the reduced spatial granularity. We did not train any models with Pvalues lower than
16 as those models have a much higher computational cost.
The Order of Space and Time Self-Attention. Our proposed “Divided Space-Time Attention” scheme applies temporal attention and spatial attention one after the other. Here,
we investigate whether reversing the order of time-space
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani