A regularization method for convolutional networks. In
Advances in Neural Information Processing Systems , pages
10750–10760, 2018. 11
[12] Golnaz Ghiasi, Tsung-Yi Lin, Ruoming Pang, and Quoc V
Le. NAS-FPN: Learning scalable feature pyramid architecture for object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2019. 8
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition , 2016.
2, 5
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Identity mappings in deep residual networks. In European
Conference on Computer Vision , 2016. 1, 2, 5, 7, 11
[15] Sepp Hochreiter and Juergen Schmidhuber. Long short-term
memory. Neural Computation , 1997. 2
[16] Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea
Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. In Advances in Neural Information
Processing Systems , pages 9423–9433, 2018. 3, 4, 5
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le