Attention Augmentation in each residual block of the last 3
stages of the architecture – when the spatial dimensions of
the activation maps are 28x28, 14x14 and 7x7 – and downsample only during the ﬁrst stage. All attention augmented
networks use =2=0.2, except for ResNet-34 which uses
==0.25. The number of attention heads is ﬁxed to Nh=8.
Architecture Params (M) Infer Train top-1
ResNet-50 25.6 - - 76.4
SE [17] 28.1 +12% +92% 77.5 (77.0)
BAM [31] 25.9 +19% +43% 77.3
CBAM [46] 28.1 +56% +132% 77.4 (77.4)
GALA [28] 29.4 +86% +133% 77.5 (77.3)
AA (= 0:25) 24.3 +29% +25% 77.7
Table 2. Image classiﬁcation performance of different attention
mechanisms on the ImageNet dataset. refers to the increase
in latency times compared to the ResNet50 on a single Tesla V100
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le