video classiﬁcation benchmarks including Kinetics 400 and
600, Epic Kitchens, Something-Something v2 and Moments
in Time, outperforming prior methods based on deep 3D
convolutional networks. To facilitate further research, we
release code at https://github.com/google-research/scenic.
1. Introduction
Approaches based on deep convolutional neural networks have advanced the state-of-the-art across many standard datasets for vision problems since AlexNet [38]. At
the same time, the most prominent architecture of choice in
sequence-to-sequence modelling (e.g. in natural language
processing) is the transformer [68], which does not use convolutions, but is based on multi-headed self-attention. This
operation is particularly effective at modelling long-range
dependencies and allows the model to attend over all elements in the input sequence. This is in stark contrast to
convolutions where the corresponding “receptive ﬁeld” is
limited, and grows linearly with the depth of the network.
The success of attention-based models in NLP has recently inspired approaches in computer vision to integrate
transformers into CNNs [75, 7], as well as some attempts to
replace convolutions completely [49, 3, 53]. However, it is
*Equal contribution
We present pure-transformer based models for video classification, drawing
upon the recent success of such models in image classification. Our model
extracts spatio-temporal tokens from the input video, which are then encoded by
a series of transformer layers. In order to handle the long sequences of tokens
encountered in video, we propose several, efficient variants of our model which
factorise the spatial- and temporal-dimensions of the input. Although
transformer-based models are known to only be effective when large training
datasets are available, we show how we can effectively regularise the model
during training and leverage pretrained image models to be able to train on
comparatively small datasets. We conduct thorough ablation studies, and achieve
state-of-the-art results on multiple video classification benchmarks including
Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in
Time, outperforming prior methods based on deep 3D convolutional networks. To
facilitate further research, we release code at
https://github.com/google-research/scenic/tree/main/scenic/projects/vivit
Anurag Arnab Mostafa Dehghani Georg Heigold Chen Sun Mario Lučić Cordelia Schmid