The posteriors over neural network weights are high dimensional and multimodal. Each mode typically characterizes a meaningfully different representation of the data. We develop Cyclical Stochastic Gradient MCMC (SG-MCMC) to automatically explore such distributions. In particular, we propose a cyclical stepsize schedule, where larger steps discover new modes, and smaller steps characterize each mode. We prove non-asymptotic convergence theory of our proposed algorithm. Moreover, we provide extensive experimental results, including ImageNet, to demonstrate the effectiveness of cyclical SG-MCMC in learning complex multimodal distributions, especially for fully Bayesian inference with modern deep neural networks.
The paper propose a new MCMC scheme which is demonstrated to perform well for estimating Bayesian neural networks. The key idea is to not keep lowering the step sizes, but -- at pre-specified times -- go back to large step sizes.
The paper is timely, the proposed algorithm is novel, and the theoretical analysis also seem quite novel.
My key concern is that with MCMC sampling it is often quite difficult to tune parameters, and by introducing more parameters to tune when step sizes should increase, I fear that we end up in a "tuning nightmare". How sensitive is the algorithm to choice of parameters?
I would expect that the proposed algorithm is quite similar to just running several MCMCs in parallel. The authors does a comparison to this and show that their approach is significantly faster due to "warm restarts". Here I wonder how sensitive this conclusion is to choice of parameters (see nightmare above) ? I would guess that opposite conclusions could be reached by tuning the algorithms differently -- is that a reasonable suspicion ?
It is argued that the cyclic nature of the algorithms gives a form of "warm start" that is beneficial for MCMC. My intuition dictate that this is only true of the modes of the posterior are reasonable close to each other; otherwise I do not see how this warm starting is helpful. I would appreciate learning more about why this intuition is apparently incorrect.
Minor comments:
* on page 4 it is stated that the proposed algorithm "automatically" provide the warm restarts -- but is it really automatic? Isn't this a priori determined by choice of parameters for the algorithm?
* It would be good to use \citet instead of \cite at places, e.g. "discussed in (Smith & Topin, 2017)" should be "discussed by Smith & Topin (2017)". This would improve readability (which is generally very good).
* For the empirical studies I think it would be good to report state-of-the-art results as well. I expect that the Bayesian nets still are subpar to non-Bayesian methods, and I think the paper should report this.