Stochastic Variational Deep Kernel Learning
Deep kernel learning combines the non-parametric flexibility of kernel methods with the inductive biases of deep learning architectures. We propose a novel deep kernel learning model and stochastic variational inference procedure which generalizes deep kernel learning approaches to enable classification, multi-task learning, additive covariance structures, and stochastic gradient training. Specifically, we apply additive base kernels to subsets of output features from deep neural architectures, and jointly learn the parameters of the base kernels and deep network through a Gaussian process marginal likelihood objective. Within this framework, we derive an efficient form of stochastic variational inference which leverages local kernel interpolation, inducing points, and structure exploiting algebra. We show improved performance over stand alone deep networks, SVMs, and state of the art scalable Gaussian processes on several classification benchmarks, including an airline delay dataset containing 6 million training points, CIFAR, and ImageNet.
This paper presents a model (SV-DKL) that use deep neural networks as feature extractors and apply additive base kernels to the output features. The deep neural networks are pre-trained followed by a training procedure of the additive base kernel and neural network parameters in cohesion. The parameters are optimized by variational inference and the learning has intriguing runtime properties. The model shows promising results compared to SVM and deep neural networks with and without GP.
This is a well-written paper with only a small number of grammatical and presentation errors, e.g. there is no numbering (a, b, c, d) in figure 2.   The theoretical basis for this work is sound and explained well. However, the experimental section lacks some detail on the training of DNNs and the pre-training. From the level of detail given it will be hard to reproduce the results.   Furthermore the results are not too convincing. The authors claim that "flexible deep kernel GPs achieve superior performance" to DNNs and SVMs. The difference in the classification results doesn't seem as convincing. It would be nice to have a more detailed discussion on whether the comparison is fair. Currently the DNNs in the stand-alone model and the DNN+GP/SV-DKL are all parameterized equally, meaning that the GP and SV-DKL additions both have more parameters. Furthermore the DNN+GP addition is trained independently, thus it will not have the same flexibility as the training of SV-DKL. Would the DNN produce better results if it were just defined with more parameters? As stated earlier it is not clear how the DNN is trained, but it would be beneficial to the training of the DNN with regularization (e.g. dropout); hence the KL term in the ELBO will work as a regularizer for the SV-DKL.  In conclusion, this is a well-written paper, but it needs to argue a better case in order to be a convincing novel contribution.