Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
We study characteristics of receptive fields of units in deep convolutional networks. The receptive field size is a crucial issue in many visual tasks, as the output must respond to large enough areas in the image to capture information about large objects. We introduce the notion of an effective receptive field, and show that it both has a Gaussian distribution and only occupies a fraction of the full theoretical receptive field. We analyze the effective receptive field in several architecture designs, and the effect of nonlinear activations, dropout, sub-sampling and skip connections on it. This leads to suggestions for ways to address its tendency to be too small.
The paper attempts to “debug” deep convolutional neural networks (CNNs) by analyzing their receptive field in the visual tasks. It provides theoretical evidence that under certain assumptions the effective receptive field (ERF) is asymptotically distributed as a Gaussian and, therefore, smaller than the theoretical receptive field (i.e., smaller than expected). The authors discuss how different blocks of CNNs affect the ERF. They suggest that performance of CNNs can be improved if the ERF is increased or grows faster width depth. Most of their theoretical findings are supported in the experimental part on image datasets (mainly CIFAR-10).
Technical quality. - The quality of experiments is overall good, but still could be improved. For instance, it is not informative to provide absolute values of classification accuracy (e.g., accuracy of 89% on CIFAR-10 is neither good for today standards, nor meaningful). Instead, a comparison should be provided in which contribution of authors ideas are clear.  Since there is such comparison (although, only in the rebuttal) for training speed clearly showing improvements, and questions of other reviewers regarding technical quality are addressed, I am able to give higher scores. The author should check if they use correlation or convolution in eq. (7)-(9) and (14). It should not be important for the distribution, but still seems that you use correlation while say that it's convolution.  Novelty/originality. The idea to analyze the effective receptive field sounds novel. It would be interesting to see analysis of more nonlinearities inherent to networks.   Potential impact or usefulness. This work potentially can be really useful and employed to improve initialization or architectures of CNNs. The authors provide some empirical evidence that making the effective receptive field larger by a special initialization method speeds up training of CNNs, which is quite valuable. The theoretical part can also be picked up and further extended by others.  Clarity and presentation. - Additional proofreading would be nice. The paper has several rather minor errors/typos. Next, all headings should be lower case (e.g., Section 4.2) according to the NIPS style.  All figures should have numbers and captions, otherwise, it can be hard to refer to them. Sentences containing equations should follow punctuation rules (e.g., equations through lines 78-82 and other instances). In several cases, additional text should be provided along with notations to ease reading of the paper. For instance, “...the receptive field size as n grows.” (line 240). So, what is ‘n’? Even though somewhere in the paper ‘n’ was denoted as something, the reader will have to search for it. - Also, few minor comments regarding references. 1) Stay consistent with proceeding titles, e.g., either use full titles as  “Advances in neural information processing systems” or abbreviated as “NIPS” for all references. 2) If the paper from arXiv you refer to was published in another (peer reviewed) place, it’s better to cite that place. For example, “Deep Residual Learning for Image Recognition” was published at CVPR.