[19] and image segmentation [6]. These observations have
inspired the research and design of new architectures, which
are typically derived from the composition of convolution
operations across an array of spatial scales and skip connections [23, 41, 39, 40, 14, 47, 13]. Indeed, automated
search strategies for designing architectures based on convolutional primitives result in state-of-the-art accuracy on
large-scale image classiﬁcation tasks that translate across a
range of tasks [55, 21].
2.2. Attention mechanisms in networks
Attention has enjoyed widespread adoption as a computational module for modeling sequences because of its
ability to capture long distance interactions [2, 44, 4, 3].
Most notably, Bahdanau et al. [2] ﬁrst proposed to combine attention with a Recurrent Neural Network [15] for
alignment in Machine Translation. Attention was further
extended by Vaswani et al. [43], where the self-attentional
Transformer architecture achieved state-of-the-art results in
Machine Translation. Using self-attention in cooperation
with convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52]. For example, the QANet [50] and Evolved Transformer [38] architectures alternate between self-attention
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le