temporal attention and spatial attention.
In Figure 3, we also compare the computational cost of joint
space-time versus divided space-time attention when using
higher spatial resolution (left) and longer (right) videos. We
note that the scheme of divided space-time scales gracefully
under both of these settings. In contrast, the scheme of
joint space-time attention leads to a dramatically higher cost
when resolution or video length is increased. In practice,
joint space-time attention causes a GPU memory overﬂow
once the spatial frame resolution reaches 448pixels, or once
the number of frames is increased to 32and thus it is effectively not applicable to large frames or long videos. Thus,
despite a larger number of parameters, divided space-time
attention is more efﬁcient than joint space-time attention
when operating on higher spatial resolution, or longer videos.
Thus, for all subsequent experiments we use a TimeSformer
constructed with divided space-time self-attention blocks.
Is Space-Time Attention All You Need for Video Understanding?
Model Pretrain K400 Training
Time (hours)K400
Acc.Inference
TFLOPsParams
I3D 8x8 R50 ImageNet-1K 444 71.0 1.11 28.0M
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani