The Wide-ResNet-28-10 architecture is comprised of 3
stages of 4 residual blocks each using two 33convolutions. We augment the Wide-ResNet-28-10 by augmenting
the ﬁrst convolution of all residual blocks with relative attention using Nh=8 heads and =2=0.2 and a minimum of
20 dimensions per head for the keys. We compare Attention
Augmentation (AA) against other forms of attention including Squeeze-and-Excitation (SE) [17] and the parameterfree formulation of Gather-Excite (GE) [16]. Table 1 shows
that Attention Augmentation improves performance both
over the baseline network and Squeeze-and-Excitation at a
similar parameter and complexity cost.
Architecture Params GFlops top-1 top-5
Wide-ResNet [51] 36.3M 10.4 80.3 95.0
GE-Wide-ResNet [16] 36.3M 10.4 79.8 95.0
SE-Wide-ResNet [17] 36.5M 10.4 81.0 95.3
AA-Wide-ResNet (ours) 36.2M 10.9 81.6 95.2
Table 1. Image classiﬁcation on the CIFAR-100 dataset [22] using
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le