Deep Learning, Computer Vision, Cnn, Transformers, AI.
ViTs varies across layers. Lower layers attend to both local and global information, while higher layers predominantly focus on global information. By analyzing the attention head mean distances for the two lowest and two highest layers in the ViT, it becomes evident that lower layers attend to a