for images that enables training of competitive fully selfattentional vision models on image classiﬁcation for the ﬁrst
time. We propose to augment convolutional operators with
this self-attention mechanism and validate the superiority ofthis approach over other attention schemes. Extensive experiments show that Attention Augmentation leads to systematic improvements on both image classiﬁcation and object detection tasks across a wide range of architectures and
computational settings.
Several open questions from this work remain. In future work, we will focus on the fully attentional regime
and explore how different attention mechanisms trade off
computational efﬁciency versus representational power. For
instance, identifying a local attention mechanism may result in an efﬁcient and scalable computational mechanism
that could prevent the need for downsampling with average
pooling [34]. Additionally, it is plausible that architectural
design choices that are well suited when exclusively relying
on convolutions are suboptimal when using self-attention
mechanisms. As such, it would be interesting to see if using Attention Augmentation as a primitive in automated architecture search procedures proves useful to ﬁnd even better models than those previously found in image classiﬁcation [55], object detection [12], image segmentation [6] and
other domains [5, 1, 35, 8]. Finally, one can ask to which
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le