temporal patterns, and thus more examples are needed by
TimeSformer to learn effectively those patterns.
4.3. Varying the Number of Tokens
The scalability of our model allows it to operate at higher
spatial resolution and on longer videos compared to most 3D
CNNs. We note that both of these aspects affect the length of
the sequence of tokens fed to the Transformer. Speciﬁcally,
increasing the spatial resolution results in a higher number
of patches (N) per frame. The number of input tokens is
also increased when using more frames. To investigate the
beneﬁts, we conduct an empirical study where we separately
increase the number of tokens along each of these two axes.
We report the ﬁndings in Figure 5. We see that increasing
the spatial resolution (up to a certain point) leads to a boost
in performance. Similarly, we observe that increasing the
length of the input clip leads to consistent accuracy gains.
Due to GPU memory constraints, we are not able to test
our model on clips longer than 96frames. Still, we would
like to point out that using clips of 96frames is a signiﬁcant
departure from current convolutional models, which are
typically limited to processing inputs of 832frames.
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani