practice, this causes a slight decrease in parameters when
replacing 3x3 convolutions and a slight increase in parameters when replacing 1x1 convolutions. Interestingly, we ﬁnd
in experiments that attention augmented networks still signiﬁcantly outperform their fully convolutional counterparts
while using less parameters.
Attention Augmented Convolutional Architectures: In
all our experiments, the augmented convolution is followed
by a batch normalization [20] layer which can learn to scale
the contribution of the convolution feature maps and the attention feature maps. We apply our augmented convolution
once per residual block similarly to other visual attention
mechanisms [17, 16, 31, 46] and along the entire architecture as memory permits (see Section 4 for more details).
Since the memory cost O((Nh(HW)2)can be prohibitive for large spatial dimensions, we augment convolutions with attention starting from the last layer (with smallest spatial dimension) until we hit memory constraints. To
reduce the memory footprint of augmented networks, we
typically resort to a smaller batch size and sometimes additionally downsample the inputs to self-attention in the layers with the largest spatial dimensions where it is applied.
Downsampling is performed by applying 3x3 average pooling with stride 2 while the following upsampling (required
for the concatenation) is obtained via bilinear interpolation.
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le