MLP, which is used to predict the ﬁnal video classes.
Space-Time Self-Attention Models. We can reduce the
computational cost by replacing the spatiotemporal attention of Eq. 5 with spatial attention within each frame only
(Eq. 6). However, such a model neglects to capture temporaldependencies across frames. As shown in our experiments,
this approach leads to degraded classiﬁcation accuracy compared to full spatiotemporal attention, especially on benchmarks where strong temporal modeling is necessary.
We propose a more efﬁcient architecture for spatiotemporal
attention, named “Divided Space-Time Attention” (denoted
with T+S), where temporal attention and spatial attention
are separately applied one after the other. This architecture
is compared to that of Space and Joint Space-Time attention
in Fig. 1. A visualization of the different attention models
on a video example is given in Fig. 2. For Divided Attention,
within each block `, we ﬁrst compute temporal attention by
comparing each patch (p;t)with all the patches at the same
spatial location in the other frames:
(`;a)time
(p;t)= SM0
@q(`;a)
(p;t)pDh>

We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani