Automated Variational Inference for Gaussian Process Models
We develop an automated variational method for approximate inference in Gaussian process (GP) models whose posteriors are often intractable. Using a mixture of Gaussians as the variational distribution, we show that (i) the variational objective and its gradients can be approximated efficiently via sampling from univariate Gaussian distributions and (ii) the gradients wrt the GP hyperparameters can be obtained analytically regardless of the model likelihood. We further propose two instances of the variational distribution whose covariance matrices can be parametrized linearly in the number of observations. These results allow gradientbased optimization to be done efficiently in a black-box manner. Our approach is thoroughly verified on five models using six benchmark datasets, performing as well as the exact or hard-coded implementations while running orders of magnitude faster than the alternative MCMC sampling approaches. Our method can be a valuable tool for practitioners and researchers to investigate new models with minimal effort in deriving model-specific inference algorithms.
I have read the authors feedback.Summary of the Paper:The paper describes a black-box variational inference method that is specific to models based on Gaussian processes. The method works by applying variational inference as the approximation method and by using a Gaussian mixture as the approximating distribution. The authors show that the lower bound in variational inference can be further lower bounded as the sum of a term that approximates the entropy of a Gaussian mixture and a cross-entropy term between the prior and the posterior approximation. A last term remains which is an expectation of the log likelihood of the model with respect to the posterior approximation. The nice thing is that the gradients of all terms except the one that depends on the likelihood can be computed in closed form. The gradient of the remaining term can be specified in terms of an expectation with respect to the posterior approximation and they can be stochastically approximated by using samples from a Gaussian distribution. In summary, the authors describe a method to carry out variational inference without the need to computed closed form expectations that may not be analytically tractable. The advantage of the method is that it computes gradients with respect to the hyper-parameters very easily. The method is compared in several experiments with deterministic VI and Markov chain Monte Carlo methods.Clarity:The paper is very well written and the use of the English language is correct. However, a few points in the paper should be further clarified. I am wondering what happens when the log of the likelihood cannot be computed. For example, this may happen when the likelihood factors are Heaviside functions of the latent functions. I also do not understand the paragraph between lines 251 and 253. It seems that the authors say that they need not use stochastic optimization methods.  However, the do approximate stochastically the gradient of the expected log likelihood. I find this statement a bit contradictory. Another point is that the authors say that they iterate until the changes in the ELBO are less than 1e-5 or the changes in the variational parameters are smaller than 1e-3. Are they evaluating the ELBO always? I think their method can be applied when this is not possible. Originality:The method described can be understood as the application of the more general method described in [16] to the specific case of Gaussian process models. The method in [16] is more general and can be applied to a range of broader models. However, for this same reason, it does not exploit the particularities of Gaussian process models. Thus, I believe the work presented here is original.Besides [16] the authors could also mention a recently published work. Namely,Michalis K. Titsias and Miguel Lazaro-Gredilla. Doubly Stochastic Variational Bayes for non-Conjugate Inference. Proc of the 31st International Conference on Machine Learning (ICML 2014)that is based on similar principles.Quality:I believe the quality of the paper is good in general and the experiments are relevant and representative for the method proposed. The same can be said about the methods the authors compare with. One thing that could be criticized is that only 5 runs are carried out and this might be insufficient to extract more conclusions from the experiments. Another thing that puzzles me is that there seems to be no clear advantage in the experiments for using a mixture of two components when compared a single component. This questions a bit the selection of that family for the posterior approximation.Significance:The experiments carried out in the paper show that the proposed method perform well in general and equivalently to other approaches based on Markov chain Monte Carlo or on deterministic variational inference. The advantage is that it needs not the computation in closed form of expectations with respect to the likelihood. These expectations are approximated by computing empirical averages with respect to a Gaussian distribution. The disadvantage is that this process is computationally more expensive, but not as expensive as Markov chain Monte Carlo methods. Again, the only point that questions a bit the significance of the proposed method is the fact that there seems to be no clear advantage of using a mixtures of Gaussians as the approximating distribution. An important point is that the authors claim advantages with respect to the method described in [16]. Specifically, they say they can very easily infer the kernel hyper-parameters, while with the method of [16] this is much more difficult. However, they do not compare with the particular approach of [16]. This questions the significance of the proposed method. A last criticism is that the authors only report gains in computation speed with respect to HMC in Section 4.4. I wonder what are the results when compared to VBO or EP. The method proposed is going to be more expensive, but by how much?Strengths:- Very well written manuscript.- Theorems to support the claims of the paper.- Simple but effective method.- Many methods compared on very different settings.Main Weaknesses:- The use of a Gaussian mixture is not supported by the experiments.- Only 5 folds are used in the experiments to compute averages.- Not comparing with respect to related methods such as [16].