Excess Risk Bounds for the Bayes Risk using Variational Inference in Latent Gaussian Models
Bayesian models are established as one of the main successful paradigms for complex problems in machine learning. To handle intractable inference, research in this area has developed new approximation methods that are fast and effective. However, theoretical analysis of the performance of such approximations is not well developed. The paper furthers such analysis by providing bounds on the excess risk of variational inference algorithms and related regularized loss minimization algorithms for a large class of latent variable models with Gaussian latent variables. We strengthen previous results for variational algorithms by showing that they are competitive with any point-estimate predictor. Unlike previous work, we provide bounds on the risk of the Bayesian predictor and not just the risk of the Gibbs predictor for the same approximate posterior. The bounds are applied in complex models including sparse Gaussian processes and correlated topic models. Theoretical results are complemented by identifying novel approximations to the Bayesian objective that attempt to minimize the risk directly. An empirical evaluation compares the variational and new algorithms shedding further light on their performance.
General comments: 
Overall this is an interesting paper with some nice theoretical results, though the techniques are rather standard. As far as I know, the results on risk bounds for Bayes predictor are novel. It provides theoretical evidence for Bayesian treating of parameters (at least for Gaussian priors) and the generalization guarantee of variational approximations (1-layer, also 2-layer by collapsing the second), which should be of wide interest to the Bayesian community. The experiments on CTMs demonstrate the risk bounds of different variational approximations, though I find the direct loss minimization case less significant because usually it’s hard to compute the marginal log likelihoods. 
The main concern that I have is the presentation of the results. It seems that the authors induce unnecessary complexity during the writing. The three kinds of risks induced in the paper ($r_{Bay}, r_{Gib}, r_{2A}$) are just negative expectations of marginal log    likelihood $\log p(y|x)$, the first term in elbo for w, and the first term in elbo for w and f (using terminology from variational inference). In Section 2.2, a 2-layer Bayesian model is introduced. I understand the authors’ intention to align with real model examples used in the paper after I finish later sections. But I still feel it’s better to derive the main results (Th.2, 6, 8) first on the 1-layer model $p(w)p(y|x,w)$ and later extend them into 2-layer settings. In fact the main theoretical result is on collapsed variational approximations, which can be seen as a 1-layer model (This is also how the proof of Corollary 3 comes).
Minor: In Section 2.2, if the 2-layer model is defined as $p(w)p(f)\prod_i p(y_i|f_i, x_i)$, should the risks be $...E_{p(f|w)} p(y|f, x)$ instead of $...E_{p(f|w,x)}p(y|f)$? I’m not sure why there is a dependence of f on x.