CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
 This paper introduces a normalization technique, which normalizes the weights of convolutional layers. The method looks like the combination of batch normalization(BN) and weight normalization. Several assumptions are provided to proceed what the authors would like to show, and some experimental results follow. pros) (+) The idea of normalizing weights looks good. cons) (-) The major problem is the experimental section, where the results are not convincing and cannot support the effectiveness of the proposed method. Specifically, the proposed method cannot outperform batch normalization (BN) in respect of validation/test accuracy even the authors claim that training is faster. It looks similar to the training curve when using smaller learning rates. (-) The proposed method (+ mixup or manifold mixup) cannot show better results over batch normalization on ImageNet dataset. (-) The proposed method seems to have a dependency on the additional method such as mixup and manifold mixup. Furthermore, the parameter of them could determine the overall performance of it. (-) Section 5.1 cannot fully support the conjecture which is that the proposed normalization method can concern the covariate shift effect. comments) - What is the benefit of the faster convergence in early epochs? We can observe faster convergence when using smaller learning rates, but finally, the larger learning rates can show better validation/test accuracy. As shown in Figure 2, and some results in Figure 4, the accuracy does now outperform BN. In this light, the faster convergence does not seem to have any advantages. - The authors should put the training curve about the first approach (shown on page 6) to clarify how the training goes differently. - For the second approach on page 6, it looks interesting the proposed method can outperform BN on CIFAR datasets when combining with mixup or manifold mixup. However, there is no analysis of why it works. - The first equation would be changed without the cyclic padding assumption, and this would cause errors normalizing all weights (just following the derived equations). So, this affects not only on edge pixels but all the pixels in features. - Updating r with "batchsize x heightout x widthout x stide^2" makes a tremendously big value in the earlier layers (i.e., close to the input), so it must affect the normalization results. The paper contains an interesting approach by normalizing weights directly, but the grounds for the conjecture are not clearly addressed. A few minor things for better readability should be addressed: all the equation should have equation number so that the reader can easily follow the paper, and the subsection titles (e.g., 4.1 Full case, 4.2 Assumptions (for what?)) are quite hard to understand what the authors will tell us.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
CONTROLLING COVARIATE SHIFT USING EQUILIBRIUM NORMALIZATION OF WEIGHTS
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output in equilibrium. We validate our method on a set of standard benchmarks including CIFAR-10/100, SVHN and ILSVRC 2012 ImageNet.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper introduces a normalization technique, which normalizes the weights of convolutional layers. The method looks like the combination of batch normalization(BN) and weight normalization. Several assumptions are provided to proceed what the authors would like to show, and some experimental results follow. pros) (+) The idea of normalizing weights looks good. cons) (-) The major problem is the experimental section, where the results are not convincing and cannot support the effectiveness of the proposed method. Specifically, the proposed method cannot outperform batch normalization (BN) in respect of validation/test accuracy even the authors claim that training is faster. It looks similar to the training curve when using smaller learning rates. (-) The proposed method (+ mixup or manifold mixup) cannot show better results over batch normalization on ImageNet dataset. (-) The proposed method seems to have a dependency on the additional method such as mixup and manifold mixup. Furthermore, the parameter of them could determine the overall performance of it. (-) Section 5.1 cannot fully support the conjecture which is that the proposed normalization method can concern the covariate shift effect. comments) - What is the benefit of the faster convergence in early epochs? We can observe faster convergence when using smaller learning rates, but finally, the larger learning rates can show better validation/test accuracy. As shown in Figure 2, and some results in Figure 4, the accuracy does now outperform BN. In this light, the faster convergence does not seem to have any advantages. - The authors should put the training curve about the first approach (shown on page 6) to clarify how the training goes differently. - For the second approach on page 6, it looks interesting the proposed method can outperform BN on CIFAR datasets when combining with mixup or manifold mixup. However, there is no analysis of why it works. - The first equation would be changed without the cyclic padding assumption, and this would cause errors normalizing all weights (just following the derived equations). So, this affects not only on edge pixels but all the pixels in features. - Updating r with "batchsize x heightout x widthout x stide^2" makes a tremendously big value in the earlier layers (i.e., close to the input), so it must affect the normalization results. The paper contains an interesting approach by normalizing weights directly, but the grounds for the conjecture are not clearly addressed. A few minor things for better readability should be addressed: all the equation should have equation number so that the reader can easily follow the paper, and the subsection titles (e.g., 4.1 Full case, 4.2 Assumptions (for what?)) are quite hard to understand what the authors will tell us.
assistant
 This paper introduces a normalization technique, which normalizes the weights of convolutional layers. The method looks like the combination of batch normalization(BN) and weight normalization. Several assumptions are provided to proceed what the authors would like to show, and some experimental results follow. pros) (+) The idea of normalizing weights looks good. cons) (-) The major problem is the experimental section, where the results are not convincing and cannot support the effectiveness of the proposed method. Specifically, the proposed method cannot outperform batch normalization (BN) in respect of validation/test accuracy even the authors claim that training is faster. It looks similar to the training curve when using smaller learning rates. (-) The proposed method (+ mixup or manifold mixup) cannot show better results over batch normalization on ImageNet dataset. (-) The proposed method seems to have a dependency on the additional method such as mixup and manifold mixup. Furthermore, the parameter of them could determine the overall performance of it. (-) Section 5.1 cannot fully support the conjecture which is that the proposed normalization method can concern the covariate shift effect. comments) - What is the benefit of the faster convergence in early epochs? We can observe faster convergence when using smaller learning rates, but finally, the larger learning rates can show better validation/test accuracy. As shown in Figure 2, and some results in Figure 4, the accuracy does now outperform BN. In this light, the faster convergence does not seem to have any advantages. - The authors should put the training curve about the first approach (shown on page 6) to clarify how the training goes differently. - For the second approach on page 6, it looks interesting the proposed method can outperform BN on CIFAR datasets when combining with mixup or manifold mixup. However, there is no analysis of why it works. - The first equation would be changed without the cyclic padding assumption, and this would cause errors normalizing all weights (just following the derived equations). So, this affects not only on edge pixels but all the pixels in features. - Updating r with "batchsize x heightout x widthout x stide^2" makes a tremendously big value in the earlier layers (i.e., close to the input), so it must affect the normalization results. The paper contains an interesting approach by normalizing weights directly, but the grounds for the conjecture are not clearly addressed. A few minor things for better readability should be addressed: all the equation should have equation number so that the reader can easily follow the paper, and the subsection titles (e.g., 4.1 Full case, 4.2 Assumptions (for what?)) are quite hard to understand what the authors will tell us.