bLVNet (Fan et al., 2019) 65.2 N/A
TimeSformer 59.5 74.9
TimeSformer-HR 62.2 78.0
TimeSformer-L 62.4 81.0
Table 7. Video-level accuracy on Something-Something-V2 and
Diving-48.Due to an issue with Diving-48 labels used in previously published results, we only compare our method with a
reproduced SlowFast 168R101 model. All models are pretained on ImageNet-1K.
about 12seconds of a Kinetics video with a single clip.
Something-Something-V2 & Diving-48. In Table 7, we
also validate our model on SSv2 and Diving-48. Since
ImageNet-21K pretraining does not improve accuracy on
SSv2 (see Table 3), in this case, we use TimeSformer pretrained on ImageNet-1K. This also allows us to apply the
same pretraining to all other models in this comparison,
using a ResNet pretrained on ImageNet-1K. Our results suggest that TimeSformer achieves lower accuracy than the best
models on this dataset. However, considering that our model
uses a completely different design, we take these results as
suggesting that TimesSformer is a promising approach even
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani