  Transformer becomes more popular in the vision domain in recent years so
there is a need for finding an effective way to interpret the Transformer model
by visualizing it. In recent work, Chefer et al. can visualize the Transformer
on vision and multi-modal tasks effectively by combining attention layers to
show the importance of each image patch. However, when applying to other
variants of Transformer such as the Swin Transformer, this method can not focus
on the predicted object. Our method, by considering the statistics of tokens in
layer normalization layers, shows a great ability to interpret the
explainability of Swin Transformer and ViT.

Computer Vision and Pattern Recognition;Artificial Intelligence