  Recently, the Vision Transformer (ViT), which applied the transformer
structure to the image classification task, has outperformed convolutional
neural networks. However, the high performance of the ViT results from
pre-training using a large-size dataset such as JFT-300M, and its dependence on
a large dataset is interpreted as due to low locality inductive bias. This
paper proposes Shifted Patch Tokenization (SPT) and Locality Self-Attention
(LSA), which effectively solve the lack of locality inductive bias and enable
it to learn from scratch even on small-size datasets. Moreover, SPT and LSA are
generic and effective add-on modules that are easily applicable to various
ViTs. Experimental results show that when both SPT and LSA were applied to the
ViTs, the performance improved by an average of 2.96% in Tiny-ImageNet, which
is a representative small-size dataset. Especially, Swin Transformer achieved
an overwhelming performance improvement of 4.08% thanks to the proposed SPT and
LSA.

Computer Vision and Pattern Recognition