Deep Learning, Computer Vision, Cnn, Transformers, AI.
Similarity between Local Attention Heads and Lower ResNet Layers: The study reveals that lower ResNet layers are most similar to the features learned by ViT’s local attention heads. When more global information is integrated, the similarity decreases monotonically, suggesting that global attention