Bayesian GAN
Generative adversarial networks (GANs) can implicitly learn rich distributions over images, audio, and data which are hard to model with an explicit likelihood. We present a practical Bayesian formulation for unsupervised and semi-supervised learning with GANs. Within this framework, we use stochastic gradient Hamiltonian Monte Carlo to marginalize the weights of the generator and discriminator networks. The resulting approach is straightforward and obtains good performance without any standard interventions such as label smoothing or mini-batch discrimination. By exploring an expressive posterior over the parameters of the generator, the Bayesian GAN avoids mode-collapse, produces interpretable and diverse candidate samples, and provides state-of-the-art quantitative results for semi-supervised learning on benchmarks including SVHN, CelebA, and CIFAR-10, outperforming DCGAN, Wasserstein GANs, and DCGAN ensembles.
over the network weights as a point mass centred on a single mode. Thus even if the generator does not memorize training examples, we would expect samples from the generator to be overly compact relative to samples from the data distribution. Moreover, each mode in the posterior over the network weights could correspond to wildly different generators, each with their own meaningful interpretations. By fully representing the posterior distribution over the parameters of both the generator and discriminator, we can more accurately model the true data distribution. The inferred data distribution can then be used for accurate and highly data-efficient semi-supervised learning.
In this paper, we propose a simple Bayesian formulation for end-to-end unsupervised and semisupervised learning with generative adversarial networks. Within this framework, we marginalize the posteriors over the weights of the generator and discriminator using stochastic gradient Hamiltonian Monte Carlo. We interpret data samples from the generator, showing exploration across several distinct modes in the generator weights. We also show data and iteration efficient learning of the true distribution. We also demonstrate state of the art semi-supervised learning performance on several benchmarks, including SVHN, MNIST, CIFAR-10, and CelebA. The simplicity of the proposed approach is one of its greatest strengths: inference is straightforward, interpretable, and stable. Indeed all of the experimental results were obtained without many of the ad-hoc techniques often used to train standard GANs.
We have made code and tutorials available at https://github.com/andrewgordonwilson/bayesgan.
Summary:
The paper introduces a Bayesian type of GAN algorithms, where the generator G and discriminator D do not have any fixed initial set of weights that gets gradually optimised. Instead, the weights for G and for D get sampled from two distributions (one for each), and it is those distributions that get iteratively updated. Different weight realisations of G may thus generate images with different styles, corresponding to different modes in the dataset. This, and the regularisation effect of the priors on the weights, promotes diversity and alleviates the mode collapse issue. The many experiments conducted in the paper support these claims.
Quality, Originality, Clarity:
While most GAN papers today mainly focus on slight variations of the original net architecture or of the original GAN objective function, this paper really presents a new, original and very interesting algorithm that combines features of GANs with a strongly Bayesian viewpoint of Neural Nets (similar in spirit, for example, to the viewpoint conveyed by Radford Neal in his PhD thesis), where the weights are parameters that should be sampled or integrated out, like any other parameter in the Bayesian framework. This leads to a very elegant algorithm. The paper contains many good experiments that prove that the algorithm is highly effective in practice (see Fig 1 and Table 1), and the produced images (though not always state of the art it seems) indeed reflect a high diversity of the samples. Moreover, the paper is very clearly written, in very good english, with almost no typos, and the algorithm is well explained. Overall: a very good paper!
Other remarks:
- Am I right in understanding that at each \theta_g sampling step, you are actually not really sampling from p(\theta_g | \theta_d), but rather from something more like p(\theta_g | \theta_d) * p(theta_d | theta_g^old) (and similar for p(\theta_d | theta_g)) ? In that case, this should be stressed and explained more clearly. In particular, I got confused at first, because I did not understand how you could sample from p(\theta_g | \theta_d) if \theta_d was not fixed.
- Algorithm 1: the inner loop over M does not stick out clearly. The 'append \theta_g^k to sample set' should be inside this loop, but seems outside. Please ensure that both inner- and outer loop are clearly visible from first eye sight.
- l.189 & l.192: clash of notations between z (of dimension d, to generate the synthetic dataset) and z (of dimension 10, as random input to the generator G). Use different variable for the first z
- Figure 1: What is p_{ML-GAN} ? I guess that the 'ML' stays for 'Maximum-Likelihood' (like the maximum-likelihood gan mentioned on l.267 without further introduction.) and that the MLGAN is just the regular GAN. In that case, please call it simply p_{GAN}, or clearly introduce the term Maximum-Likelihood GAN and its abbreviation MLGAN (with a short justification of why it is a maximum likelihood, which would actually be an interesting remark).
- l.162-171: Describe the architectures more carefully. Anyone should be able to implement *exactly* your architecture to verify the experiments. So sentences like ('The networs we use are slightly different from [10] (e.g. we have 4 hidden layers and fewer filters per layer)', without saying exactly how many filters, is not ok. Also, the architecture for the synthetic dataset is described partly at l.162-163 and partly at l.191. Please group the description of the network in one place. The authors may consider describing roughly their architectures in the main part, and put a detailed description in the appendix.
Response to rebuttal:
We thank the authors for answering our questions. We are happy to hear that the code will be released (as there is usually no better way to actually reproduce the experiments with the same parameter settings). Please enclose the short explanations of the rebuttal on GANs being the Maximum Likelihood solution with improper priors into your final version.