Understanding Probabilistic Sparse Gaussian Process Approximations Mark van der Wilk †
Good sparse approximations are essential for practical inference in Gaussian Processes as the computational cost of exact methods is prohibitive for large datasets. The Fully Independent Training Conditional (FITC) and the Variational Free Energy (VFE) approximations are two recent popular methods. Despite superficial similarities, these approximations have surprisingly different theoretical properties and behave differently in practice. We thoroughly investigate the two methods for regression both analytically and through illustrative examples, and draw conclusions to guide practical application.
The paper gives theoretical analysis of FITC and FVE approximations for sparse large-scale Gaussian processes. There are various analytical results and proofs that are interesting.
The paper analyses the differences between the two main large-scale approximations strategies for GP inference, namely the variational inference of Titsias'09, and the FITC by Snelson'06.  It's an interesting paper that tries to summarise how these two main approaches differ and how they affect the resulting models. The paper is limited in that it only covers how these two approximation methods affect the resulting models, but does not comment on the running times at all. The only motivation for sparse approximations is large-scale problems, and thus scalability. Perhaps for this reason they do not consider e.g. EP and SVI approaches, which have their own interesting approximation effects the resulting model as well. A longer format might fit this paper better, with analyses of all currently used approaches instead of just these two.  All the analyses are interesting, especially regarding the local minima, heteroscedastic noises and addition of inducing points. 3.4. illustrates beautifully the behaviors of FITC/FVE. It seems that FVE is superior, but then in 3.5. a real dataset shows FITC matching or surpassing VFE performance. To explain this discrepancy more large-scale datasets should have been analysed to see if FITC consistently beats VFE, and in which settings. The theoretical drawbacks of FITC would be cast in strange light if it does not seem to suffer from them in practise, as hinted in table 2.  I'm also having problems with the running example regarding FITC. Fig1 shows the FITC to give a badly "pinched" model. The same dataset was used in many other papers (with roughly comparable experimental settings), where this does not happen for FITC. In both Snelson'06 and Titsias'09 the FITC gives a good non-pinched model for this data, and in Quinonero'05 the FITC gives slighly pinched model, but not as bad as in the present paper. This is likely due to number of datapoints (fig3 again is not pinched), so it would be nice to see how the "pinching" effects behaves as a function of data size.  Overall it's an excellent paper with good findings on FITC/FVE, but the theory and practise seem to disagree in large-scale cases, and there is no running time analysis. The paper would also be greatly strenghtened by being more comprehensive in compared methods (eg. EP/SVI).