The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
It has been noted in existing literature that over-parameterization in ReLU networks generally improves performance. While there could be several factors involved behind this, we prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, it is known that He initialization in deep ReLU networks asymptotically preserves variance of activations in the forward pass and variance of gradients in the backward pass for infinitely wide networks, thus preserving the flow of information in both directions. Our paper goes beyond these results and shows novel properties that hold under He initialization: i) the norm of hidden activation of each layer is equal to the norm of the input, and, ii) the norm of weight gradient of each layer is equal to the product of norm of the input vector and the error at output layer. These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to be larger than a certain finite lower bound. As we show, this lower bound depends on the depth of the network and the number of samples, and by the virtue of being a lower bound, over-parameterized ReLU networks are endowed with these desirable properties. For the aforementioned hidden activation norm property under He initialization, we further extend our theory and show that this property holds for a finite width network even when the number of data samples is infinite. Thus we overcome several limitations of existing papers, and show new properties of deep ReLU networks at initialization.
This paper studies initialization techniques for deep ReLU networks from a theoretical standpoint and derives finite layer width concentration bounds to show that with the He initialization scheme, deep ReLU networks preserve the norm of the input sample during a forward pass and the norm of the gradient with respect to the output during a backward pass. The concentration bounds also suggest lower bounds on the width of the ReLU layers. The authors verify their theory with experiments on synthetic data. While I believe the finite sample concentration bounds for networks initialized using the He initialization are valuable, it seems to me that this work is incremental in terms of understanding the He initialization. The techniques used in the paper are also fairly well known Chernoff bounding techniques and concentration of Gaussian random vectors and matrices. The authors claims on explaining overparameterization are also overstated in my opinion. While the authors are able to obtain a lower bound on layer widths in order to preserve norms during forward/backward passes at initialization, the lower bound is only dependent on the input dimension and not the size of the dataset, which is the relevant quantity to decide whether a model is under/over parameterized. Even in the authors* bounds for finite datasets, what I can surmise from their results (it would be better to explicitly state it if that is one of the goals of this paper) is that the width of each layer needs to be atleast log(N) where N is the size of the dataset. This is hardly overparameterized. Furthermore, the authors do not explain how studying the properties of the initialization might help understand generalization at minima. Since gradient descent based techniques seem to prefer solutions that are close to initialization, the analysis in this paper might be a useful starting point in understanding generalization. The authors could also consider how adding BatchNorm layers and/or Residual connections affect the He initialization scheme, and whether initialization matters for those techniques. Finite width concentration bounds for initialization of networks using Batchnorm/Residual connections could be useful. To summarize, I do not see how the authors claims about explaining overparameterization (even at initialization) can be made. Without those claims the contribution of this paper is incremental and does not warrant publication at this time. I am willing to adjust my score if the claims about overparameterization are stated explicitly and make sense.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
It has been noted in existing literature that over-parameterization in ReLU networks generally improves performance. While there could be several factors involved behind this, we prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, it is known that He initialization in deep ReLU networks asymptotically preserves variance of activations in the forward pass and variance of gradients in the backward pass for infinitely wide networks, thus preserving the flow of information in both directions. Our paper goes beyond these results and shows novel properties that hold under He initialization: i) the norm of hidden activation of each layer is equal to the norm of the input, and, ii) the norm of weight gradient of each layer is equal to the product of norm of the input vector and the error at output layer. These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to be larger than a certain finite lower bound. As we show, this lower bound depends on the depth of the network and the number of samples, and by the virtue of being a lower bound, over-parameterized ReLU networks are endowed with these desirable properties. For the aforementioned hidden activation norm property under He initialization, we further extend our theory and show that this property holds for a finite width network even when the number of data samples is infinite. Thus we overcome several limitations of existing papers, and show new properties of deep ReLU networks at initialization.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper studies initialization techniques for deep ReLU networks from a theoretical standpoint and derives finite layer width concentration bounds to show that with the He initialization scheme, deep ReLU networks preserve the norm of the input sample during a forward pass and the norm of the gradient with respect to the output during a backward pass. The concentration bounds also suggest lower bounds on the width of the ReLU layers. The authors verify their theory with experiments on synthetic data. While I believe the finite sample concentration bounds for networks initialized using the He initialization are valuable, it seems to me that this work is incremental in terms of understanding the He initialization. The techniques used in the paper are also fairly well known Chernoff bounding techniques and concentration of Gaussian random vectors and matrices. The authors claims on explaining overparameterization are also overstated in my opinion. While the authors are able to obtain a lower bound on layer widths in order to preserve norms during forward/backward passes at initialization, the lower bound is only dependent on the input dimension and not the size of the dataset, which is the relevant quantity to decide whether a model is under/over parameterized. Even in the authors* bounds for finite datasets, what I can surmise from their results (it would be better to explicitly state it if that is one of the goals of this paper) is that the width of each layer needs to be atleast log(N) where N is the size of the dataset. This is hardly overparameterized. Furthermore, the authors do not explain how studying the properties of the initialization might help understand generalization at minima. Since gradient descent based techniques seem to prefer solutions that are close to initialization, the analysis in this paper might be a useful starting point in understanding generalization. The authors could also consider how adding BatchNorm layers and/or Residual connections affect the He initialization scheme, and whether initialization matters for those techniques. Finite width concentration bounds for initialization of networks using Batchnorm/Residual connections could be useful. To summarize, I do not see how the authors claims about explaining overparameterization (even at initialization) can be made. Without those claims the contribution of this paper is incremental and does not warrant publication at this time. I am willing to adjust my score if the claims about overparameterization are stated explicitly and make sense.
assistant
This paper studies initialization techniques for deep ReLU networks from a theoretical standpoint and derives finite layer width concentration bounds to show that with the He initialization scheme, deep ReLU networks preserve the norm of the input sample during a forward pass and the norm of the gradient with respect to the output during a backward pass. The concentration bounds also suggest lower bounds on the width of the ReLU layers. The authors verify their theory with experiments on synthetic data. While I believe the finite sample concentration bounds for networks initialized using the He initialization are valuable, it seems to me that this work is incremental in terms of understanding the He initialization. The techniques used in the paper are also fairly well known Chernoff bounding techniques and concentration of Gaussian random vectors and matrices. The authors claims on explaining overparameterization are also overstated in my opinion. While the authors are able to obtain a lower bound on layer widths in order to preserve norms during forward/backward passes at initialization, the lower bound is only dependent on the input dimension and not the size of the dataset, which is the relevant quantity to decide whether a model is under/over parameterized. Even in the authors* bounds for finite datasets, what I can surmise from their results (it would be better to explicitly state it if that is one of the goals of this paper) is that the width of each layer needs to be atleast log(N) where N is the size of the dataset. This is hardly overparameterized. Furthermore, the authors do not explain how studying the properties of the initialization might help understand generalization at minima. Since gradient descent based techniques seem to prefer solutions that are close to initialization, the analysis in this paper might be a useful starting point in understanding generalization. The authors could also consider how adding BatchNorm layers and/or Residual connections affect the He initialization scheme, and whether initialization matters for those techniques. Finite width concentration bounds for initialization of networks using Batchnorm/Residual connections could be useful. To summarize, I do not see how the authors claims about explaining overparameterization (even at initialization) can be made. Without those claims the contribution of this paper is incremental and does not warrant publication at this time. I am willing to adjust my score if the claims about overparameterization are stated explicitly and make sense.