In this paper, we present an in-depth investigation of the convolutional autoencoder (CAE) bottleneck. Autoencoders (AE), and especially their convolutional variants, play a vital role in the current deep learning toolbox. Researchers and practitioners employ CAEs for a variety of tasks, ranging from outlier detection and compression to transfer and representation learning. Despite their widespread adoption, we have limited insight into how the bottleneck shape impacts the emergent properties of the CAE. We demonstrate that increased height and width of the bottleneck drastically improves generalization, which in turn leads to better performance of the latent codes in downstream transfer learning tasks. The number of channels in the bottleneck, on the other hand, is secondary in importance. Furthermore, we show empirically that, contrary to popular belief, CAEs do not learn to copy their input, even when the bottleneck has the same number of neurons as there are pixels in the input. Copying does not occur, despite training the CAE for 1,000 epochs on a tiny (≈ 600 images) dataset. We believe that the findings in this paper are directly applicable and will lead to improvements in models that rely on CAEs.
Summary:
This paper studies some of the properties of fully convolutional autoencoders (CAE) as a function of the shape and total size of the bottleneck. They train and test CAEs with bottlenecks consisting of different ratios of spatial resolution versus number of channels, as well as different total number of neurons. The authors investigate which type of change in the bottleneck is most influential on training behavior, generalization to test set, and linear separability for classification/regression. Their first main finding is that the spatial resolution of the bottleneck is a stronger influencer of generalization to the test set than the number of channels and the total number of neurons in the bottleneck. The second main finding is that even when the total number of neurons in the bottleneck is equal to the data input size, the neural network does not appear to simply learn to copy the input image into the bottleneck. 
Decision: 
Weak reject: It is always refreshing to see papers that address/challenge/investigate common assumptions in deep learning. However, I find the experimental findings and discussion of borderline quality for a full conference paper. It might be more suitable as a good workshop paper.
Supporting arguments for decision:
It is unclear to me why the authors have chosen to only take a subset of the CelebA and STL-10 datasets for training and testing. It seems like dataset size is also an important factor that increases the complexity for training a model, and it certainly affects how a model can generalize. When auto-encoders are studied in other literature it is uncommon practice to restrict the dataset sizes like this, so this makes me question the applicability of this paper’s results to the literature. 
It seems that the experimental validation is based on one run per CAE model with one single seed. This is on the low side of things, especially when quite extensive claims are made. An example of such a claim is on page 6 when discussing a sudden jump in training and test scores for 8x8x48 model trained on the Pokemon dataset. Because the same behavior appeared when the authors repeated the experiment with the same seed, the authors conclude “This outlier suggests, that the loss landscape might not always be as smooth towards the end of training, as some publications (Goodfellow et al., 2014) claim and that ‘cliffs” (i. e., sudden changes in loss) can occur even late in training.” Making this claim based on something that occurs with one single model for a single seed is not convincing and overstating this finding. 
Another example is on page 7 under bullet point 4, where the authors discuss the obtained evidence against copying behaviour when the bottleneck is of the same size as the input. The authors state “We believe this finding to have far-reaching consequences as it directly contradicts the popular hypothesis about copying CAEs.” The paper definitely shows some empirical evidence that supports the claim that copying does not occur, but these findings are all done with a single seed and by considering small subsets of datasets (celebA and stl-10). In my opinion, it is therefore too much to state that the current findings have far reaching consequences. It has potential, but I wouldn’t go much further than that. 
On page 7 in the second to last paragraph the influence of dataset complexity is discussed. The authors state “the loss curves and reconstruction samples do not appear to reflect the notion of dataset difficulty we defined in Section 2.3” and “This lack of correspondence implies that the intuitive and neural network definitions of difficulty do not align. Nevertheless, a more detailed study is required to answer this question definitively as curriculum learning research that suggests the opposite (Bengio et al., 2009) also exists.” It is unclear to me what the authors expected to find here. Moreover, the absence of major differences across the chosen datasets does not immediately make me doubt or question results from curriculum learning. My skepticism is again enhanced by the fact that the authors have taken a subset of the data for the more complex celebA and STL-10 datasets. Dataset size seems like a crucial part of dataset complexity.
An interesting smaller finding is that linear separability of the latent codes for classification is better on the test set for the pokemon dataset, even though the training and test reconstruction losses showed signs of overfitting. The authors hypothesize that overfitting might occur more in the decoder than in the encoder.
Additional feedback to improve the paper (not part of decision assessment)
- Section 2.3.1: what was the original resolution of the pokemon dataset?
- Section 3.1, c_j is used as the compression level, but in eq 3 and 4 c^i is also used to indicate the number of channels. - For clarity please use n_ci in eq 2 and 3 or choose a different all together for the compression level.
- Please increase the font size of the plots in figures 1, 3 and 4.