Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.
Review of “Principled Weight Initialization for Hypernetworks”
There has been a lot of existing work on neural network initialization, and much of this work has made large impact in making deep learning models easier to train in practice. There has also been a line of work on indirect encoding of neural works (i.e. HyperNEAT work of Stanley, and more recent Hypernetworks proposed by Ha et al) which showed promising results of training very large networks (in the case of Stanley), or have network weights that can adapt to the training data (in the case of Hypernetworks), and these approaches have been shown to be useful in applications such as meta-learning or few-shot learning (i.e. [1]). However, as far as I know, there hasn't been any work that looks at a principled way of initializing the weights of a weight-generating network, which this work tries to explore.
Making the observation (and claim) that traditional init methods don't init hypernetworks properly, they propose a few techniques to initialize hypernetworks ("Hyperfan"-family), which are justified in a similar way as original classical init techniques (i.e. preserving variance like in Xavier init), and they demonstrate that their method works well for feed forward networks on MNIST, CIFAR-10 tasks compared to traditional classical init methods, as well for a continual learning task.
I liked the paper as they identified a problem that hasn't been studied, and proposed a reasonable method to solve it. Their method may be able to make Hypernetworks accessible to many more researchers and practitioners, the way classifical init techniques have made neural net training more accessible.
There are a few things that could improve the paper (and get an improvement score from me). The authors don't have to do all of these, but just a few suggestions:
1) The experiments, to my understanding, are all feed forward networks. How about RNNs or LSTMs?
2) Are there any (interesting) tasks that use Hypernetworks that are not trainable with existing methods, but made trainable using this proposed scheme?
3) Would this method also work with HyperNEAT [2] or Compressed Network Search [3]? (probably should cite that line of work too). In [3], a research group at IDSIA used DCT compression to compress millions of weights into a few dozen parameters, so would be interesting if the approach will work on similar "learn-from-pixels" RL experiments.
I'm assigning a score of 6 (it's currently like a "really good" workshop paper, but a normal conf paper IMO), but I like this paper and would like to see the authors make an attempt to improve it, so I can improve the score to see it get accepted with a higher certainty.
Good luck!
[1] i.e. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6519722/ https://arxiv.org/pdf/1710.03641.pdf https://arxiv.org/pdf/1703.03400.pdf
[2] http://eplex.cs.ucf.edu/hyperNEATpage/
[3] http://people.idsia.ch/~juergen/compressednetworksearch.html
*** Revised Score ***
Nov20: Upon reading the other reviews, and looking at the changes to the paper with the extra citations, I'm improving the score to 8. (For the record, if this was a 1-10 scale, I would have liked my score to be a 7).