Supervised autoencoders: Improving generalization performance with unsupervised regularizers
Generalization performance is a central goal in machine learning, with explicit generalization strategies needed when training over-parametrized models, like large neural networks. There is growing interest in using multiple, potentially auxiliary tasks, as one strategy towards this goal. In this work, we theoretically and empirically analyze one such model, called a supervised auto-encoder: a neural network that jointly predicts targets and inputs (reconstruction). We provide a novel generalization result for linear auto-encoders, proving uniform stability based on the inclusion of the reconstruction error-particularly as an improvement on simplistic regularization such as norms. We then demonstrate empirically that, across an array of architectures with a different number of hidden units and activation functions, the supervised auto-encoder compared to the corresponding standard neural network never harms performance and can improve generalization.
This paper performs an analysis of linear supervised autoencoders, showing that they have nice O(1/t) stability properties that in turn transfers to a nice generalization error bound. They also show that in practice, regularizing a deep model with an autoencoder penalty outperforms L2 and dropout regularizations in some scenarios. This is most notable if the model is allowed more capacity: the capacity is used to fit the reconstruction (with apparent benefits) rather than overfit on the predictive task.  This result seems to be an interesting improvement over Liu et al. if one cares about/believes in reconstruction. While I think most in our community would agree that capturing factors of variation through reconstruction is very appealing (and previous literature favorably points that way), there are instances (unfortunately rarely documented) of auxiliary reconstruction losses harming performance or requiring extensive tuning, and the bound that you compute does not necessarily suggest a better performance either. I intuitively agree that reconstruction seems preferable to L2, and as you say, “potentially without strongly biasing the solution”, but I would not be surprised if under certain mild conditions the bias that a reconstruction loss adds was demonstrably larger than that of L2.  The empirical part nicely demonstrates the point of the authors (and has a good methodology). Something feels like it is missing though. The main result of this paper is a O(1/t) bound over t the number of dataset samples, yet, there is not a single plot that reflects varying the size of a dataset.  The paper itself was well written, and although I had to read a few of the referenced papers to get familiar with stability bounds, the paper felt clear. I think overall this is a neat contribution that could maybe be stronger but is nonetheless relevant to deep learning training understanding.  I gave myself a confidence score of 3: while I think I understand the paper and its results, I do not have sufficient understanding of the theory to say with certainty that the math is 100% correct.  Minor comments: The equation following (2) has a typo: the first L_p should be a function of x’_m and y’_p,m (rather than x’_i and y’_p,i). wrt Theorem 1. In practice, people rarely bound their parameters (except maybe for RNNs, and even then people usually only clip gradients). Without L2, what stops say B_F or rather ||F|| from growing in O(t)? (Maybe this is wrong since this is for the linear case, but since more data points can require a more complex hypothesis they might require larger weights) CIFAR-10 is a specific subset of 60k examples of a larger dataset (the tiny images dataset), which itself has 80 million examples. I’d either correct the number or explicit what CIFAR-10 is, because I assume that when you “used a random subset of 50,000 images for training and 10,000 images for testing” you only shuffled the actual 60k image dataset, but referencing the 80M images suggests otherwise. It’s great that you do cross-validation. I think mentioning meta-parameter ranges in the appendix would be useful for reproducibility (or the optimal ones you’ve found). I think you could easily do a plot that’s exactly like Figure 3, but with the x axis being the number of samples used, from say 1 to 50k   Rebuttal update: I thank the authors for their response. Again I think this could be a great paper and encourage the authors to add varieties of experiments that would make this work even more convincing.