AA-ResNet-34 Relative (ours) 74.7 92.0
AA-ResNet-50 None 77.5 93.7
AA-ResNet-50 2d Sine 77.5 93.7
AA-ResNet-50 CoordConv 77.5 93.8
AA-ResNet-50 Relative (ours) 77.7 93.8
Table 7. Effects of different position encodings in Attention Augmentation on ImageNet classiﬁcation.
Position Encodings mAP COCO mAP 50 mAP 75
None 37.7 56.0 40.2
CoordConv [29] 37.4 55.5 40.1
Relative (ours) 38.2 56.5 40.7
Table 8. Effects of different position encodings in Attention Augmentation on the COCO object detection task using a RetinaNet
AA-ResNet-50 backbone.
relative position encodings to other position encoding
schemes. We apply Attention Augmentation using the same
hyperparameters as 4.2 with the following different position encoding schemes: 1) The position-unaware version of
self-attention (referred to as None ), 2) a two-dimensional
implementation of the sinusoidal positional waves (referred
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le