Deep Learning, Artificial Intelligence, Image Classification, Vision Transformer, Resnet.
(small distances) and global heads (large distances). This is in contrast to CNNs, which are hardcoded to attend only locally in the lower layers. At higher layers, all self-attention heads are global. Effective Receptive Field Lower layer effective receptive fields for ViT are indeed larger than