Attention operators have been widely applied on data of various orders and dimensions such as texts, images, and videos. One challenge of applying attention operators is the excessive usage of computational resources. This is due to the usage of dot product and softmax operator when computing similarity scores. In this work, we propose the Siamese similarity function that uses a feed-forward network to compute similarity scores. This results in the Siamese attention operator (SAO). In particular, SAO leads to a dramatic reduction in the requirement of computational resources. Experimental results show that our SAO can save 94% memory usage and speed up the computation by a factor of 58 compared to the regular attention operator. The computational advantage of SAO is even larger on higher-order and higher-dimensional data. Results on image classification and restoration tasks demonstrate that networks with SAOs are as effective as models with regular attention operator, while significantly outperform those without attention operators.
The authors introduce a novel self-attention operator for neural networks. Their self-attention operator computes similarity between elements a and b as (a+b)^Tw where w is a learned parameter and does not use the softmax operator. This leads to improvements in space and time complexity compared to regular self-attention which uses the dot product (a^Tb).
They show that concatenating their operation with convolution brings improvements over the MobileNetv2 baseline on ImageNet classification and over U-Net on restoration tasks.
Attention has been empirically shown to bring improvements in many visual tasks but certain methods (such as self-attention) can be quite expensive in computations and memory. Identifying cheaper attention mechanisms that obtain similar accuracy performance as expensive attention mechanisms is therefore an important direction for work.
However, I take several crucial issues with this work and especially the evaluation/presentation of the methods:
- Although this is the focus of the work, the authors do not report actual memory consumption and latency times for MobileNetv2, SANet and the other attention mechanisms. (There is no need for the simulated scenarios of Table 2 since we already know the theoretical complexities of the different methods).
- The authors only compare their methods against regular self-attention (without or with pooling/softmax), and ignore a longstanding literature of other (potentially cheaper in terms of memory and computations) attention mechanisms in vision (see below). Without comparison to at least Squeeze-and-Excite, it is hard to evaluate the significance of the method presented in the draft.
- The motivation for naming the method "siamese" is quite poor. Siamese networks typically are more complex than a single layer feed forward (which is just a dot-product). Furthermore, the siamese similarity (as introduced by the authors) does not respect the usual properties of similarity functions. For example siasim(a, 0) = a^tw = 1/2 siasim(a,a) can take arbitrary values including negative values ("a can be dissimilar with itself")
- X vs Q, K, V? Self-attention is incorrectly described as "a special case of the attention operator with Q = K = V", instead of Q = XW^Q, K=XW^K, V = XW^V.
- In Table 7, shouldn't SANet w/o params have less params than SANet?
In summary, the paper addresses an important challenge and proposes a technically sound method.  However, the current  draft has fundamental experimental flaws in its evaluation/presentation and lacks comparison against relevant cheap channelwise attention mechanisms (such as Squeeze-and-Excitation). I argue for rejection.
Relevant literature:
- channelwise attention: Squeeze-and-Excitation, Gather-Excite
- Channelwise and spatial attention: Bottleneck Attention Module, Convolutional Block Attention Module
- Relative Self-attention for vision: Attention Augmented Convolutional Networks, An Empirical Study of Spatial Attention Mechanisms in Deep Networks.