  Many of our core assumptions about how neural networks operate remain
empirically untested. One common assumption is that convolutional neural
networks need to be stable to small translations and deformations to solve
image recognition tasks. For many years, this stability was baked into CNN
architectures by incorporating interleaved pooling layers. Recently, however,
interleaved pooling has largely been abandoned. This raises a number of
questions: Are our intuitions about deformation stability right at all? Is it
important? Is pooling necessary for deformation invariance? If not, how is
deformation invariance achieved in its absence? In this work, we rigorously
test these questions, and find that deformation stability in convolutional
networks is more nuanced than it first appears: (1) Deformation invariance is
not a binary property, but rather that different tasks require different
degrees of deformation stability at different layers. (2) Deformation stability
is not a fixed property of a network and is heavily adjusted over the course of
training, largely through the smoothness of the convolutional filters. (3)
Interleaved pooling layers are neither necessary nor sufficient for achieving
the optimal form of deformation stability for natural image classification. (4)
Pooling confers too much deformation stability for image classification at
initialization, and during training, networks have to learn to counteract this
inductive bias. Together, these findings provide new insights into the role of
interleaved pooling and deformation invariance in CNNs, and demonstrate the
importance of rigorous empirical testing of even our most basic assumptions
about the working of neural networks.

Computer Vision and Pattern Recognition;Machine Learning;Machine Learning