SE-ResNet-101 [17] 15.6 49.3M 78.4 94.2
AA-ResNet-101 (ours) 16.1 45.4M 78.7 94.4
ResNet-152 [14] 23.0 60.2M 78.4 94.2
SE-ResNet-152 [17] 23.1 66.8M 78.9 94.5
AA-ResNet-152 (ours) 23.8 61.6M 79.1 94.6
Table 3. Image classiﬁcation on the ImageNet dataset [9] across
a range of ResNet architectures: ResNet-34, ResNet-50, Resnet101, and ResNet-152 [14, 47, 13].
Table 2 benchmarks Attention Augmentation against
channel and spatial attention mechanisms BAM [31],
CBAM [46] and GALA [28] with channel reduction ratio= 16 on the ResNet50 architecture. Despite the
2We found that augmenting the pointwise expansions works just as well
but does not save parameters or computations.
Architecture GFlops Params top-1 top-5
MnasNet-0.75 0.45 2.91M 73.3 91.3
AA-MnasNet-0.75 0.51 3.02M 73.9 91.6
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le