PAC-Bayesian Theory Meets Bayesian Inference
We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the Bayesian marginal likelihood. That is, for the negative log-likelihood loss function, we show that the minimization of PAC-Bayesian generalization risk bounds maximizes the Bayesian marginal likelihood. This provides an alternative explanation to the Bayesian Occam's razor criteria, under the assumption that the data is generated by an i.i.d. distribution. Moreover, as the negative log-likelihood is an unbounded loss function, we motivate and propose a PAC-Bayesian theorem tailored for the sub-gamma loss family, and we show that our approach is sound on classical Bayesian linear regression tasks.
The paper presents (long desired) links between PAC-Bayesian theory and Bayesian inference, based on treating the negative log-likelihood as a loss function and the resultant bounds show a trade-off between the KL-divergence between the prior and posterior and the NLL. Technically this involves extending the normal analysis that only applies to bounded losses to the real line. Secondly, it is shown that Bayesian model selection through the model evidence can be linked to minimizing the PAC-Bayes bound. They show how this is applied to the special case of Bayesian linear regression.
The paper is well written and theoretically strong. It's been conjectured in the past that there should be links between PAC-Bayes theory and Bayesian inference, but to my knowledge this is the first theoretically complete demonstration of such links.  Some comments:  - In eq(8) (and above) the notion of a prior with bounded likelihood is introduced. Am I right in thinking that this is a data-dependent prior, since it can only be known if the likelihood will be bounded for a given prior after observing the data? If this is not the case can you explain how such a prior is possible?  - I note that Theorem 3 is from unpublished work, although the proof is given as A.1. However I'm not sure the proof in A.1 is complete: Firstly \zeta_\pi and \Phi_{\ell, \pi, D} are not defined. Secondly, what are the requirements on X' and Y'?   - in the proof of 14/15: -- where does the first inequality come from? -- it would be nice to state that you used the Chi-squared MGF during the proof, as it took me a while to work this out -- I couldn't quite work out why the equality was only valid for \lambda < \frac{1}{c}. Again I think this relates to the chi-squared MGF?   Minor comments: - I'm not sure it helps to have so many references in the footnotes on p1. Perhaps if a numbered citation format was used rather than author-year, they could be put into the text - L22  do you mean implicitly rather than explicitly? - L95  Theorems 1 -> Theorem 1 - L185  mode -> model - L203  bounds -> bound - L206  is this L now the number of hyperparameters rather than the number of models? - L229 can be express by -> can be expressed as - L247 to a Gaussian -> by a Gaussian - Figure 1 panel (b) - legend is hard to read