models on this dataset. However, considering that our model
uses a completely different design, we take these results as
suggesting that TimesSformer is a promising approach even
for challenging temporally-heavy datasets, such as SSv2.
In Table 7, we also present our method on another
“temporally-heavy” dataset, Diving-48. Due to a recently
discovered issue with a previous version of Diving-48 labels,
here, we only compare our method with a reproduced SlowFast168R101 model. Our results show that TimeSformer
outperforms SlowFast by a substantial margin.
4.6. Long-Term Video Modeling
Lastly, we evaluate TimeSformer on the task of long-term
video modeling using HowTo100M (Miech et al., 2019).
HowTo100M is an instructional video dataset that contains
around 1M instructional Web videos showing humans performing over 23K different tasks, such as cooking, repairing,
making arts, etc. The average duration of these videos is
around 7minutes, which is orders of magnitude longer than
the duration of videos in standard action recognition benchmarks. Each HowTo100M video has a label indicating the
task demonstrated in the video (one out of the 23K classes),
which can be used for supervised training. Thus, it is a good
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani