  The recently proposed Visual image Transformers (ViT) with pure attention
have achieved promising performance on image recognition tasks, such as image
classification. However, the routine of the current ViT model is to maintain a
full-length patch sequence during inference, which is redundant and lacks
hierarchical representation. To this end, we propose a Hierarchical Visual
Transformer (HVT) which progressively pools visual tokens to shrink the
sequence length and hence reduces the computational cost, analogous to the
feature maps downsampling in Convolutional Neural Networks (CNNs). It brings a
great benefit that we can increase the model capacity by scaling dimensions of
depth/width/resolution/patch size without introducing extra computational
complexity due to the reduced sequence length. Moreover, we empirically find
that the average pooled visual tokens contain more discriminative information
than the single class token. To demonstrate the improved scalability of our
HVT, we conduct extensive experiments on the image classification task. With
comparable FLOPs, our HVT outperforms the competitive baselines on ImageNet and
CIFAR-100 datasets. Code is available at https://github.com/MonashAI/HVT

Computer Vision and Pattern Recognition