We introduce HyperGAN, a generative network that learns to generate all the weight parameters of deep neural networks. HyperGAN first transforms low dimensional noise into a latent space, which can be sampled from to obtain diverse, performant sets of parameters for a target architecture. We utilize an architecture that bears resemblance to generative adversarial networks, but we evaluate the likelihood of samples with a classification loss. This is equivalent to minimizing the KL-divergence between the generated network parameter distribution and an unknown true parameter distribution. We apply HyperGAN to classification, showing that HyperGAN can learn to generate parameters which solve the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters. We also show that HyperGAN can also provide better uncertainty than standard ensembles. This is evaluated by the ability of HyperGAN-generated ensembles to detect out of distribution data as well as adversarial examples. We see that in addition to being highly accurate on inlier data, HyperGAN can provide reasonable uncertainty estimates.
This paper proposes a technique for learning a distribution over parameters of a neural network such that samples from the distribution correspond to performant networks. The approach effectively encourages sampled parameters to have low loss on the training set, and also uses an adversarial loss to encourage the distribution of parameters to be Gaussian distributed. This approach can improve performance slightly by using ensembling and can be useful for uncertainty estimates for out-of-distribution examples. The approach is tested on a few simple problems and is shown to work well.
I am definitely in favor of exploring adversarial divergences (using a critic as a differentiable loss to compare two distributions) in unusual settings, and this paper certainly does this. The idea of transforming samples from a prior such that the transformed sample corresponds to useful network parameters is interesting. The results also seem promising. However, currently the mathematical description of this method is completely unclear and ridden with many errors. I can understand at a reasonable level what the approach is doing from Figure 1, but the definitions and equations given in Equation 3 are at times nearly incomprehensible "mathiness". I'm giving the paper a borderline accept because the idea is interesting and the results are OK; I will raise my score if Section 3 is dramatically improved. I give some specific examples of issues with Section 3 in my specific comments below. I'd also note that the paper does a somewhat poor job comparing to existing work - only section 4.2 includes a comparison to existing "uncertainty" methods. This should also be improved - the authors should implement the existing methods and use them as a point of comparison in all of their experiments. As a final high-level note, the approach is described at various points as an "autoencoder" particularly in reference to the adversarial autoencoder. However, the approach does not "autoencode" anything - there is no reconstruction term, or input apart from the noise samples. The only thing it has in common with the adversarial autoencoder is the use of a critic to enforce a distributional constraint. Calling it, or comparing it to, an autoencoder is confusing and misleading.
Specific comments:
- You mention fast weights in related work. I believe Hinton and Plaut were the first to propose fast weights in "Using Fast Weights to Deblur Old Memories", and I'd also suggest mentioning "Using Fast Weights to Attend to the Recent Past" which is a more recent demonstration that fast weights can be useful on modern problems.
- The are some issues with your description of Equation 1: First, I don't believe you define G(z) (I assume it is the "decoder" network; please define it). Second, in practice I don't believe you actually use JSD or MMD for D_z; you use a critic architecture which in some limit approximates some statistical divergence but in practice they typically don't (see e.g. Arjovsky and Bottou 2017; Fedus et al. 2017; Rosca et al. 2018). Third, writing Q_z \sim Q(z | x) seems strange to me - Q_z is a distribution, and I don't believe that Q(z | x) is a distribution over distributions, so how are you sampling a distribution (Q_z) from Q(z | x) as suggested by the use of the \sim notation? I think you simply mean that Q_z is Q(z | x) approximately marginalized over x.
- Equation 2 is also not clear. First, the sentence before starts "Suppose the real parameters \theta^* \sim \Theta..." The equation itself does not include \theta^* or \Theta so I don't see what this is referring to. Second, the expression for an m-dimensional is written \mathcal{N}(0, \sigma^2, I_m). It's not clear why there is a comma before I_m, and I_m is not defined (though I assume it is the m \times m identity matrix) - did you mean to multiply I_m by \sigma^2? Third, it looks like you actually define P_z twice, once as "an m-dimensional isotropic Gaussian" and again as "a Kd-dimensional isotropic Gaussian"; am I to infer that m = Kd? Why use both? Fourth, you mention the joint P(x, y) but the expectation is taken over P_x and P(y | x). Why call it P_x and not P(x)? And why not compute the expectation over P(x, y)? Fifth, you write "Here the encoder..." -- you never define that Q(z) or G is "the encoder", I assume Q(z). It is strange to take the expectation over Q(z) (I assume sampling z \sim Q(z)) but then have the term Q(z) appear in (2). How are Q(z) and Q_z related? On that note, I don't see how (2) is an autoencoder, since there is no Q(z | x) term. It appears instead that you are sampling z from Q(z) which doesn't condition on x. So what is being autoencoded? Related, you write "all the q_k (that will generate different layers) will be correlated, unlike dimensions of z which are drawn to be independent from each other." But if Q(z) = [q_1, ..., q_K] then doesn't the secont term in (2) suggest that they are being enforced to be similar to the prior P_z, and therefore uncorrelated? Note that you also say later on "The job of the regularizer D_z(P_z, Q_z) is to force each embedding q_n to approximate P_z." Frankly at this point I will stop pointing out issues with this equation and discussion since they are so widespread.
- In your definition of your ensemble scoring rule, you are taking the sum over N + 1 elements (n = 0 to N) but dividing by N.
- In 4.2, do you use the same model architecture/training/regularization etc. as in previous studies? If not I think comparing the different methods will be conflated by differences in training procedures. Since you do not report results in many experimental settings, I assume you don't.
- In Figure 3, why not plot the true standard deviation around the true function? It appears you are only plotting +/- 3 stndard deviations for the learned function.
- Why not include 100 models L2 on Figure 3?
- It's not clear to me why you define your "disagreement d" when it appears the same as the entropy score you used in 4.4.
- A stronger and more convincing attack would be to attack the ensemble of models, instead of attacking a single model and testing on the ensemble.