Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
While the use of bottom-up local operators in convolutional neural networks (CNNs) matches well some of the statistics of natural images, it may also prevent such models from capturing contextual long-range feature interactions. In this work, we propose a simple, lightweight approach for better context exploitation in CNNs. We do so by introducing a pair of operators: gather, which efficiently aggregates feature responses from a large spatial extent, and excite, which redistributes the pooled information to local features. The operators are cheap, both in terms of number of added parameters and computational complexity, and can be integrated directly in existing architectures to improve their performance. Experiments on several datasets show that gather-excite can bring benefits comparable to increasing the depth of a CNN at a fraction of the cost. For example, we find ResNet-50 with gather-excite operators is able to outperform its 101-layer counterpart on ImageNet with no additional learnable parameters. We also propose a parametric gather-excite operator pair which yields further performance gains, relate it to the recently-introduced Squeeze-and-Excitation Networks, and analyse the effects of these changes to the CNN feature activation statistics.
This paper proposes to augment standard deep residual networks for image classification with explicit gather-scatter operations, in order to improve propagation of useful contextual information.  It implements a per-channel gather-scatter operation.  The gather phase performs either global average pooling or max pooling (termed "picking") per channel, over an extended spatial domain.  The scatter phase upsamples the gathered feature maps back to the original size, and combines them with the input feature maps.  Experiments show integrating gather-scatter operations into residual networks yields a performance boost on the standard ImageNet classification task without any, or with minimal, parameter increase.  The paper justifiably discusses connections with Squeeze-and-Excitation networks [10].  However, it is missing citation, discussion, and comparison to two other highly related recent publications:  (1) Multigrid Neural Architectures     Tsung-Wei Ke, Michael Maire, Stella X. Yu     CVPR 2017  (2) Non-local Neural Networks     Xiaolong Wang, Ross Girshick, Abhinav Gupta, Kaiming He     CVPR 2018  The first of these works, [Ke et al, CVPR 2017] extends residual networks with a scale-space information propagation mechanism: at each layer, information flows both up and down a scale-space pyramid.  This is implemented by combining convolution operations with max-pooling, upsampling, and concatenation at every layer in the network.  These ingredients sound quite similar to the proposed gather-scatter scheme and, in fact, could be viewed as a generalization of the proposed gather-scatter approach to one which operates at all spatial scales.  Like the current paper, [Ke et al] specifically mention rapid integration of contextual cues and demonstrate improved accuracy and parameter savings on ImageNet classification using their multigrid residual network.  The second of these related works, [Wang et al, CVPR 2018], suggests inserting a non-local information gathering operation into neural networks, similar in formulation to the gather operation proposed here.  They discuss multiple functional forms and also present results of experiments modifying residual networks to include their proposed operations.  The existence of both of these published prior works decreases the relative novelty of the approach proposed here.  The fact that neither are cited is a serious problem.  Ideally, both would be cited, discussed, and compared against in experiments.  If the particular implementation proposed by this paper has advantages, a thorough experimental comparison to prior work is needed to justify them.  In light of this prior work, I do not believe the idea or current results alone are sufficiently novel or impressive to carry the paper.  ---  The rebuttal addresses my concerns about discussion of related work and addition of experimental comparison to that work. I have raised my overall score.