Principled Weight Initialization for Hypernetworks
Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.
Principled Weight Initialization for Hypernetworks Summary: This paper investigates initialization schemes for hypernets, models where the weights are generated by another neural network, which takes as input either a learned embedding or the activations of the model itself. Standard initialization schemes for deep networks (Glorot, He, Fixup) are based on variance analyses that aim to keep the scale of activations/gradients well-behaved through forward-backward propagation, but using this approach is ill-founded for hypernets where the output is a set of weights rather than e.g. a softmax’d classification output. This paper extends the standard variance analysis to consider the hypernet case by investigating what the choice of hypernet initialization should be if one still wishes to maintain the well-behaved activations/gradients in the main model. The authors present results showing the evolution of hypernet outputs with their scheme are better-behaved, demonstrate that their modification leads to improved stability for hypernet-based convnets on CIFAR (over a model which, for the standard choice of He init, basically does not train) and improved performance on a continual learning task. My take: This is an *aha!* or an “obvious in retrospect” paper: a simple idea based on noticing something being done wrong in practice with a fairly straightforward fix, coupled with a decent empirical evaluation and analysis. The paper is well-written and reasonably easy to follow (although I am not familiar with Ricci calculus I did not feel that I was flummoxed at any point during the maths), and the potential impact is decent: any future work which employs a hypernetwork would likely do well to consider the methods in this paper. While I would like to see the empirical evaluations taken a bit further, I think this paper is a solid accept (7/10) and would make a good addition to ICLR2020. Notes: While this is a good paper, I think the impact of the paper could be magnified if the authors were a bit more ambitious with their empirical evaluations. This method seems to enable training hypernets in settings which would previously have been unstable; it would be good to more thoroughly characterize robustness to hyperparameters or otherwise demonstrate additional practical value. Showing results on ImageNet would also be helpful (I’m well aware this is not always in the realm of compute possibility) or just showing progress on a more challenging task outside of CIFAR or a small continual learning problem would, I think, greatly increase the chances that this paper catches on. As this is a “unleash your potential” note, I have not taken this sentiment into account in my review (as should hopefully be evident from my accept score). Minor: -Caption for figure 2 should indicate what kind of magnitudes are represented—are these the average weight norms in each layer vs epochs? The axes should be labeled.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Principled Weight Initialization for Hypernetworks
Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Principled Weight Initialization for Hypernetworks Summary: This paper investigates initialization schemes for hypernets, models where the weights are generated by another neural network, which takes as input either a learned embedding or the activations of the model itself. Standard initialization schemes for deep networks (Glorot, He, Fixup) are based on variance analyses that aim to keep the scale of activations/gradients well-behaved through forward-backward propagation, but using this approach is ill-founded for hypernets where the output is a set of weights rather than e.g. a softmax’d classification output. This paper extends the standard variance analysis to consider the hypernet case by investigating what the choice of hypernet initialization should be if one still wishes to maintain the well-behaved activations/gradients in the main model. The authors present results showing the evolution of hypernet outputs with their scheme are better-behaved, demonstrate that their modification leads to improved stability for hypernet-based convnets on CIFAR (over a model which, for the standard choice of He init, basically does not train) and improved performance on a continual learning task. My take: This is an *aha!* or an “obvious in retrospect” paper: a simple idea based on noticing something being done wrong in practice with a fairly straightforward fix, coupled with a decent empirical evaluation and analysis. The paper is well-written and reasonably easy to follow (although I am not familiar with Ricci calculus I did not feel that I was flummoxed at any point during the maths), and the potential impact is decent: any future work which employs a hypernetwork would likely do well to consider the methods in this paper. While I would like to see the empirical evaluations taken a bit further, I think this paper is a solid accept (7/10) and would make a good addition to ICLR2020. Notes: While this is a good paper, I think the impact of the paper could be magnified if the authors were a bit more ambitious with their empirical evaluations. This method seems to enable training hypernets in settings which would previously have been unstable; it would be good to more thoroughly characterize robustness to hyperparameters or otherwise demonstrate additional practical value. Showing results on ImageNet would also be helpful (I’m well aware this is not always in the realm of compute possibility) or just showing progress on a more challenging task outside of CIFAR or a small continual learning problem would, I think, greatly increase the chances that this paper catches on. As this is a “unleash your potential” note, I have not taken this sentiment into account in my review (as should hopefully be evident from my accept score). Minor: -Caption for figure 2 should indicate what kind of magnitudes are represented—are these the average weight norms in each layer vs epochs? The axes should be labeled.
assistant
Principled Weight Initialization for Hypernetworks Summary: This paper investigates initialization schemes for hypernets, models where the weights are generated by another neural network, which takes as input either a learned embedding or the activations of the model itself. Standard initialization schemes for deep networks (Glorot, He, Fixup) are based on variance analyses that aim to keep the scale of activations/gradients well-behaved through forward-backward propagation, but using this approach is ill-founded for hypernets where the output is a set of weights rather than e.g. a softmax’d classification output. This paper extends the standard variance analysis to consider the hypernet case by investigating what the choice of hypernet initialization should be if one still wishes to maintain the well-behaved activations/gradients in the main model. The authors present results showing the evolution of hypernet outputs with their scheme are better-behaved, demonstrate that their modification leads to improved stability for hypernet-based convnets on CIFAR (over a model which, for the standard choice of He init, basically does not train) and improved performance on a continual learning task. My take: This is an *aha! * or an “obvious in retrospect” paper: a simple idea based on noticing something being done wrong in practice with a fairly straightforward fix, coupled with a decent empirical evaluation and analysis. The paper is well-written and reasonably easy to follow (although I am not familiar with Ricci calculus I did not feel that I was flummoxed at any point during the maths), and the potential impact is decent: any future work which employs a hypernetwork would likely do well to consider the methods in this paper. While I would like to see the empirical evaluations taken a bit further, I think this paper is a solid accept (7/10) and would make a good addition to ICLR2020. Notes: While this is a good paper, I think the impact of the paper could be magnified if the authors were a bit more ambitious with their empirical evaluations. This method seems to enable training hypernets in settings which would previously have been unstable; it would be good to more thoroughly characterize robustness to hyperparameters or otherwise demonstrate additional practical value. Showing results on ImageNet would also be helpful (I’m well aware this is not always in the realm of compute possibility) or just showing progress on a more challenging task outside of CIFAR or a small continual learning problem would, I think, greatly increase the chances that this paper catches on. As this is a “unleash your potential” note, I have not taken this sentiment into account in my review (as should hopefully be evident from my accept score). Minor: -Caption for figure 2 should indicate what kind of magnitudes are represented—are these the average weight norms in each layer vs epochs? The axes should be labeled.