ﬂops efﬁcient (see Table 6).
4We consider pointwise convolutions as dense layers. This architecture
employs 4 non-pointwise convolutions in the stem and the ﬁrst stage of the
architecture, but we believe such operations can be replaced by attention
too.Architecture GFlops Params top-1 top-5
ResNet-34 [14] 7.4 21.8M 73.6 91.5
ResNet-50 [14] 8.2 25.6M 76.4 93.1
== 0:25 7.9 24.3M 77.7 93.8
== 0:5 7.3 22.3M 77.3 93.6
== 0:75 6.8 20.7M 76.7 93.2
== 1:0 6.3 19.4M 73.9 91.5
Table 6. Attention Augmented ResNet-50 with varying ratios of
attentional channels.
0 25 50 75 100
fraction of attention versus convolutional channels707274767880top-1 accuracy76.4%77.5%
76.6%
75.4%
71.1%77.7%
77.3%
76.7%
73.9%
Convolutional networks have been the paradigm of choice in many computer
vision applications. The convolution operation however has a significant
weakness in that it only operates on a local neighborhood, thus missing global
information. Self-attention, on the other hand, has emerged as a recent advance
to capture long range interactions, but has mostly been applied to sequence
modeling and generative modeling tasks. In this paper, we consider the use of
self-attention for discriminative visual tasks as an alternative to
convolutions. We introduce a novel two-dimensional relative self-attention
mechanism that proves competitive in replacing convolutions as a stand-alone
computational primitive for image classification. We find in control
experiments that the best results are obtained when combining both convolutions
and self-attention. We therefore propose to augment convolutional operators
with this self-attention mechanism by concatenating convolutional feature maps
with a set of feature maps produced via self-attention. Extensive experiments
show that Attention Augmentation leads to consistent improvements in image
classification on ImageNet and object detection on COCO across many different
models and scales, including ResNets and a state-of-the art mobile constrained
network, while keeping the number of parameters similar. In particular, our
method achieves a $1.3\%$ top-1 accuracy improvement on ImageNet classification
over a ResNet50 baseline and outperforms other attention mechanisms for images
such as Squeeze-and-Excitation. It also achieves an improvement of 1.4 mAP in
COCO Object Detection on top of a RetinaNet baseline.
Irwan Bello Barret Zoph Ashish Vaswani Jonathon Shlens Quoc V. Le