Deep Learning, Computer Vision, Cnn, Transformers, AI.
mix of local and global information, whereas higher layers attend solely to global information. ViT’s Local Attention Heads and ResNet Layers: By comparing subsets of ViT attention heads (ranging from the most locally attending heads to the most globally attending heads) to the lowest layer