Bayesian neural networks, which both use the negative log-likelihood loss function and average their predictions using a learned posterior over the parameters, have been used successfully across many scientific fields, partly due to their ability to 'effortlessly' extract desired representations from many large-scale datasets. However, generalization bounds for this setting is still missing. In this paper, we present a new PAC-Bayesian generalization bound for the negative log-likelihood loss which utilizes the Herbst Argument for the log-Sobolev inequality to bound the moment generating function of the learners risk. We explore the generalization and calibration properties of the learned posterior on several image classification benchmarks, showing that the proposed approach provides better generalization and uncertainty estimates.
Summary:
This paper proposes a PAC-Bayesian generalization bound for Bayesian neural networks. The author discuss that earlier generalization bounds hold for bounded loss function, whereas the proposed generalization bound considers negative log likelihood loss which can be unbounded. Therefore, earlier results do not hold here. 
The technical approach used is along lines of PAC Bayes framework and specifically for this loss function which requires bounding the log-partition function, the authors follow the Herbst Argument for bounding the log-partition function. 
Contribution: 
The paper uses straight forward PAC Bayes approach and the only bottleneck is bounding the log-partition function for which the authors use an earlier result (Herbst Argument for bounding the log-partition function. )
Significance: 
My biggest concern with this work is its significance. As we know classification loss is bounded and for regression loss, as long as we have bounded input and a Lipschitz function corresponding to the NN, the output is bounded. Also as authors mention, there have been two earlier results covering other unbounded loss functions. Therefore, I do not feel that extending those results to NLL is a good enough contribution. Especially since the extension uses a known approach  (Herbst Argument for bounding the log-partition function. )
Experiments: 
* From the explanations, it seems each architecture is trained once, which is not acceptable. How can one refute the effect of a specific initial value? A good scientific practice entails having mean and variance bar for different values or at least repeating the experiment multiple times and reporting the avg. 
* According to the paper, the architectures in table 2, fig 1 are made by keeping the number of parameters roughly the same. Then the authors increase the depth. Note that to keep the # of parameters the same, they have to decrease the width as they increase the depth. Therefore, this cannot be a correct analysis of effect of the depth. As depth is not the only parameter that is changed between the architectures.
Writing: 
The writing is overall ok which some vague places such as 
*first page, last paragraph, line 1: ".. our PAC Bayesian bound is tightest when we learn..." the authors do not discuss what is the space of options for the bound and only mention the case when it is tightest. Therefore the claim is confusing
*first page, last paragraph, last line: "..better uncertainty than the baseline". The authors do not specify the baseline which makes this whole claim vague.
Title: 
This is a minor issue but worth mentioning. The title is vague and confusing. In the first read one might think this paper provides PAC-Bayesian bounds for usual NNs (which has been considered and written about many times in the literature). The authors should mention that the considered networks are Bayesian NNs.