Z., Wu, Y ., and Hughes, M. The best of both worlds:
Combining recent advances in neural machine translation.
InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics . Association for
Computational Linguistics, 2018a.
Chen, Y ., Kalantidis, Y ., Li, J., Yan, S., and Feng, J. Aˆ2nets: Double attention networks. In Advances in Neural
Information Processing Systems 31 , 2018b.
Chen, Y ., Fan, H., Xu, B., Yan, Z., Kalantidis, Y ., Rohrbach,
M., Yan, S., and Feng, J. Drop an octave: Reducing
spatial redundancy in convolutional neural networks with
octave convolution. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV) ,
October 2019.
Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. CoRR ,
2019.
Cordonnier, J., Loukas, A., and Jaggi, M. On the relationship between self-attention and convolutional layers. In
8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30,
We present a convolution-free approach to video classification built
exclusively on self-attention over space and time. Our method, named
"TimeSformer," adapts the standard Transformer architecture to video by
enabling spatiotemporal feature learning directly from a sequence of
frame-level patches. Our experimental study compares different self-attention
schemes and suggests that "divided attention," where temporal attention and
spatial attention are separately applied within each block, leads to the best
video classification accuracy among the design choices considered. Despite the
radically new design, TimeSformer achieves state-of-the-art results on several
action recognition benchmarks, including the best reported accuracy on
Kinetics-400 and Kinetics-600. Finally, compared to 3D convolutional networks,
our model is faster to train, it can achieve dramatically higher test
efficiency (at a small drop in accuracy), and it can also be applied to much
longer video clips (over one minute long). Code and models are available at:
https://github.com/facebookresearch/TimeSformer.
Gedas Bertasius Heng Wang Lorenzo Torresani