Learning Invariances using the Marginal Likelihood
Generalising well in supervised learning tasks relies on correctly extrapolating the training data to a large region of the input space. One way to achieve this is to constrain the predictions to be invariant to transformations of the input that are known to be irrelevant (e.g. translation). Commonly, this is done through data augmentation, where the training set is enlarged by applying hand-crafted transformations to the inputs. We argue that invariances should instead be incorporated in the model structure, and learned using the marginal likelihood, which correctly rewards the reduced complexity of invariant models. We demonstrate this for Gaussian process models, due to the ease with which their marginal likelihood can be estimated. Our main contribution is a variational inference scheme for Gaussian processes containing invariances described by a sampling procedure. We learn the sampling procedure by backpropagating through it to maximise the marginal likelihood.
The paper deals with incorporating invariances in Gaussian processes. Specifically, the authors propose a method for making Gaussian process models less sensitive to transformations (affine transformations, rotations etc). In their set-up, the transformations under consideration are described by a probability distribution and the invariance is achieve by averaging the kernel of the Gaussian process wrt. this distribution of transformations.   For this class of models, both the kernel of invariant functions as well as the posterior distributions of interest are intractable. However, the authors deal with both of these issues using the so-called inter-domain inducing points approach. One major advantage of the proposed method is that the only requirement for the distribution of the transformations is that it can be sampled from. Importantly, the proposed method also allows learning parameters of the transformations from data through the marginal likelihood. The drawback is that they are only able to derive lower bounds for the marginal likelihood of the invariant model for Gaussian likelihoods and logistic likelihoods.  The paper is concluded by applying the proposed method to the MNIST dataset and some of its variations.    The submission appears technically sound and correct. The claims of the paper are not supported by any theoretical analysis, but the paper does contain experimental results for the proposed method showing the benefits of incorporating invariance.  My prime concern is that the only included datasets are MNIST and variations of MNIST and yet, their results still seem quite sensitive. It would be a significant improvement, if the authors could demonstrate the benefit of the model on more complex datasets. Also, it would be interesting if the authors would compare their method against data augmentation.  The paper is in general well-structured, well-written and easy to follow.  Their idea for incorporating invariance in Gaussian processes in original and novel.  Incorporating invariance into probabilistic models is an important problem. Despite the fact that experimental results could be more convincing and it is not possible to do multi-class classification in a proper probabilistic fashion, I still believe that the results are important and that other researchers will most likely build upon and improve their method.    Further comments: Line 26: “...likelihood cannot be used to compare to other model.” typo  Line 23, 36: The following two sentences are contradictions: “Data augmentation is also unsatisfactory from a Bayesian perspective” (line 23) and “We believe this work to be exciting, as it simultaneously provides a justification for data augmentation from a Bayesian point of view” (line 36)   Line 134: “Data augmentation attempts to do the same, only by considering a general set points for each input x.” typo  Line 189: “The intractability stems less the bound than the approximate posterior q(f ), …” Something seems to be missing here   Line 228: How many samples S is needed in practice?  Line 234: “For example, the logistic likelihood can be written as an expectation over a Pòlya-Gamma variable ω [REF]”. Missing reference   Line 275: “However, all invariant models outperform a GP model with simple RBF kernel by a large margin both in terms of error and marginal likelihood (not shown), see Fig. 2 (right).” If the right-most panel does not show the marginal likelihood, then what does it show?  After author feedback ----------------------------- I've read the rebuttal, but it has not changed my opinion: it is a good submission with significant contributions to the field.