Many of our core assumptions about how neural networks operate remain empirically untested. One common assumption is that convolutional neural networks need to be stable to small translations and deformations to solve image recognition tasks. For many years, this stability was baked into CNN architectures by incorporating interleaved pooling layers. Recently, however, interleaved pooling has largely been abandoned. This raises a number of questions: Are our intuitions about deformation stability right at all? Is it important? Is pooling necessary for deformation invariance? If not, how is deformation invariance achieved in its absence? In this work, we rigorously test these questions, and find that deformation stability in convolutional networks is more nuanced than it first appears: (1) Deformation invariance is not a binary property, but rather that different tasks require different degrees of deformation stability at different layers. (2) Deformation stability is not a fixed property of a network and is heavily adjusted over the course of training, largely through the smoothness of the convolutional filters. (3) Interleaved pooling layers are neither necessary nor sufficient for achieving the optimal form of deformation stability for natural image classification. (4) Pooling confers too much deformation stability for image classification at initialization, and during training, networks have to learn to counteract this inductive bias. Together, these findings provide new insights into the role of interleaved pooling and deformation invariance in CNNs, and demonstrate the importance of rigorous empirical testing of even our most basic assumptions about the working of neural networks.
It is often argued that one of the roles of pooling is to increase the stability of neural networks to 
deformations. This paper presents empirical evidence to contest this assertion, or at least qualify it.
I appreciate empirical studies that question some of the widely accepted dogmas of deep learning. 
From this point of view, the present paper is certainly interesting.
Unfortunately, the actual evidence presented is quite weak, and insufficient to draw far reaching 
conclusions. An obvious objection is the authors only consider two datasets, and a very small number of 
more or less standard pooling methodologies. The effect of pooling is evaluated in terms of cosine 
similarlity, which is not necessarily a good proxy for the actual performance of a network.
A more serious issue is that they seem to very readily jump to unwarranted conclusions. For example, 
the fact that stability to deformations (by which I necessarily mean the specific type of deformations 
that they consider) tends to decrease in the middle layers of neural networks during training does not 
mean that starting with a neural network with less stability would be better. Maybe some kind of 
spontaneous coarse-to-fine optimization is going on in the network. Similarly, it is obvious that smoother 
filters are going to lead to more stable representations. However, they might be less good at discriminative 
tasks. Just because smoother filters are more stable does not automatically mean that they are more desirable.
Stability to deformations is an important but subtle topic in computer vision. For starters, it is difficult 
to define what kind of deformations one wants to be insensitive to in the first place. A useful model would 
likely incorporate some notion of deformations at multiple different length scales. 
Just showing that one network is better than another wrt some arbitrarily defined simple class of deformations 
with no reference to actual recognition performance, speed of training, or interpretation of the nature of 
the deformations and the learned filters is not very convincing. I would particularly like to emphasize the 
last point. I would really like to understand what pooling actually does, not just at the level of "if you 
turn it off, then cosine similarity will decrease by this much or that much."