“tubes” from the input volume, and to linearly project this to
Rd. This method is an extension of ViT’s embedding to 3D,
and corresponds to a 3D convolution. For a tubelet of dimensionthw,nt=bT
tc,nh=bH
hcandnw=bW
wc,
tokens are extracted from the temporal, height, and width
dimensions respectively. Smaller tubelet dimensions thus
result in more tokens which increases the computation.
Intuitively, this method fuses spatio-temporal information
during tokenisation, in contrast to “Uniform frame sampling” where temporal information from different frames is
fused by the transformer.
3.3. Transformer Models for Video
As illustrated in Fig. 1, we propose multiple transformerbased architectures. We begin with a straightforward extension of ViT [18] that models pairwise interactions between all spatio-temporal tokens, and then develop more
efﬁcient variants which factorise the spatial and temporal
dimensions of the input video at various levels of the transformer architecture.
Model 1: Spatio-temporal attention This model simply forwards all spatio-temporal tokens extracted from the
video, z0, through the transformer encoder. We note that
this has also been explored concurrently by [4] in their
We present pure-transformer based models for video classification, drawing
upon the recent success of such models in image classification. Our model
extracts spatio-temporal tokens from the input video, which are then encoded by
a series of transformer layers. In order to handle the long sequences of tokens
encountered in video, we propose several, efficient variants of our model which
factorise the spatial- and temporal-dimensions of the input. Although
transformer-based models are known to only be effective when large training
datasets are available, we show how we can effectively regularise the model
during training and leverage pretrained image models to be able to train on
comparatively small datasets. We conduct thorough ablation studies, and achieve
state-of-the-art results on multiple video classification benchmarks including
Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in
Time, outperforming prior methods based on deep 3D convolutional networks. To
facilitate further research, we release code at
https://github.com/google-research/scenic/tree/main/scenic/projects/vivit
Anurag Arnab Mostafa Dehghani Georg Heigold Chen Sun Mario Lučić Cordelia Schmid