Modern convolutional networks are not shift-invariant, despite their convolutional nature: small shifts in the input can cause drastic changes in the internal feature maps and output. In this paper, we isolate the cause -the downsampling operation in convolutional and pooling layers -and apply the appropriate signal processing fix -low-pass filtering before downsampling. This simple architectural modification boosts the shift-equivariance of the internal representations and consequently, shift-invariance of the output. Importantly, this is achieved while maintaining downstream classification performance. In addition, incorporating the inductive bias of shift-invariance largely removes the need for shift-based data augmentation. Lastly, we observe that the modification induces spatially-smoother learned convolutional kernels. Our results suggest that this classical signal processing technique has a place in modern deep networks.
This work shows that adding a simple blurring into max pooling layers can address issues of image classification instability under small image shifts. In general this work presents a simple and easy to implement solution to a common problem of CNNs and even though it lacks more thorough theoretical analysis of this problem from the signal processing perspective (such as minimal size of the blurring kernel for fulfilling the Nyquist-Shannon sampling theorem), it seems to provide ample empirical evidence.
Pros:
+ The introduction and motivation is really well written and Figure 3 provides a clear visualisation main max pooling operator issues.
+ The proposed method is really simple and shows promising results on the CIFAR dataset. With random shifts, authors had to tackle cropping with circular shifts. As it can cause artifacts in the data, authors also provide baseline performances on the original data (used for both training and testing).
+ Authors provide a thorough evaluation, ranging from comparing hidden representations to defining consistency metrics of the classified classes.
This work is lacking in the experimental section due to some missing details and few inconsistencies. I believe the most of my concerns can be relatively easily fixed/clarified in an update of this submission.
Major issues, which if fixed would improve the rating:
- It is not correct to average test accuracy and test consistency as both measures are different quantities, especially when using them for ranking. The difference between accuracy of different methods are considerably smaller than differences in the classification consistency. 
- It is not clear how many shifts are used for computing the "Random Test Accuracy" and the "Classification Accuracy". Also whether the random shifts are kept constant between evaluated networks and evaluation metrics.
- Authors do not address the question what is the correct order of operations for the blurring. E.g. would the method empirically work if blurring was applied before max pooling? Do the operations commute?
- The selection of the filters is rather arbitrary, especially regarding the 1D FIR filters. The separability of these filters should be discussed.
- I believe authors should address how this work differs to [1], as it also tests different windowing functions for pooling operators, even though in different tasks.
Minor issues, which would be nice to fix however which do not influence my rating:
* Section 3.1 - And L-Layer deep *CNN*, H_l x W x C_l -> H_l x W_l x C_l
* Section 3.1. Last paragraph - I would not agree with the statement that in CNNs the shift invariance must necessarily emerge upon shift equivariance. If anything, this may hold only for the last layer of a network without fully connected layers and with average pooling of the classifier output (ResNet/GoogleNet like networks).
* Explicitly provide the network architecture as [Simonyan14] does not test on CIFAR and cannot use Batch normalisation.
* It would be useful to add citation for the selected FIR filters.
* The flow of section 4.2. can be improved to help readability. The three metrics should be first motivated before their introduction. Metric 2. paragraph - the metric is defined below, not above. 
* It would be interesting to see what would be the performance if the blurring filters were trained as well (given some sensible initialisation).
* One future direction would be to verify that this approach generalises to larger networks as well. It might be worth to discuss this in the conclusions.
[1] Scherer, Dominik, Andreas Müller, and Sven Behnke. "Evaluation of pooling operations in convolutional architectures for object recognition." Artificial Neural Networks–ICANN 2010. Springer, Berlin, Heidelberg, 2010. 92-101.