  Vision Transformers have attracted a lot of attention recently since the
successful implementation of Vision Transformer (ViT) on vision tasks. With
vision Transformers, specifically the multi-head self-attention modules,
networks can capture long-term dependencies inherently. However, these
attention modules normally need to be trained on large datasets, and vision
Transformers show inferior performance on small datasets when training from
scratch compared with widely dominant backbones like ResNets. Note that the
Transformer model was first proposed for natural language processing, which
carries denser information than natural images. To boost the performance of
vision Transformers on small datasets, this paper proposes to explicitly
increase the input information density in the frequency domain. Specifically,
we introduce selecting channels by calculating the channel-wise heatmaps in the
frequency domain using Discrete Cosine Transform (DCT), reducing the size of
input while keeping most information and hence increasing the information
density. As a result, 25% fewer channels are kept while better performance is
achieved compared with previous work. Extensive experiments demonstrate the
effectiveness of the proposed approach on five small-scale datasets, including
CIFAR-10/100, SVHN, Flowers-102, and Tiny ImageNet. The accuracy has been
boosted up to 17.05% with Swin and Focal Transformers. Codes are available at
https://github.com/xiangyu8/DenseVT.

Computer Vision and Pattern Recognition