Batch Normalization has Multiple Benefits: An Empirical Study on Residual Networks
Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples within a batch, performs poorly when the batch size is too small, and significantly increases the cost of computing a parameter update in some models. This work identifies two practical benefits of batch normalization. First, it improves the final test accuracy. Second, it enables efficient training with larger batches and larger learning rates. However we demonstrate that the increase in the largest stable learning rate does not explain why the final test accuracy is increased under a finite epoch budget. Furthermore, we show that the gap in test accuracy between residual networks with and without batch normalization can be dramatically reduced by improving the initialization scheme. We introduce “ZeroInit”, which trains a 1000 layer deep Wide-ResNet without normalization to 94.3% test accuracy on CIFAR-10 in 200 epochs at batch size 64. This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large. We also show that ZeroInit matches the validation accuracy of batch normalization when training ResNet-50-V2 on ImageNet at batch size 1024.
This paper conducts extensive experiments to study batch normalization, a very popular technique for training a deep convolutional network and its relationship with learning rate and batch size. In addition, the authors also propose a new initialization scheme, “ZeroInit”, to train a deep ResNet for better test accuracy. This is a very empirical study and the authors also show extensive experimental results. However, I do not see any novel findings in this study. Mostly this paper confirms the results of previous studies. The experimental results do not show much advantage of ZeroInit either. Overall, it is unclear what is the major novel contribution in this paper.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Batch Normalization has Multiple Benefits: An Empirical Study on Residual Networks
Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples within a batch, performs poorly when the batch size is too small, and significantly increases the cost of computing a parameter update in some models. This work identifies two practical benefits of batch normalization. First, it improves the final test accuracy. Second, it enables efficient training with larger batches and larger learning rates. However we demonstrate that the increase in the largest stable learning rate does not explain why the final test accuracy is increased under a finite epoch budget. Furthermore, we show that the gap in test accuracy between residual networks with and without batch normalization can be dramatically reduced by improving the initialization scheme. We introduce “ZeroInit”, which trains a 1000 layer deep Wide-ResNet without normalization to 94.3% test accuracy on CIFAR-10 in 200 epochs at batch size 64. This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large. We also show that ZeroInit matches the validation accuracy of batch normalization when training ResNet-50-V2 on ImageNet at batch size 1024.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper conducts extensive experiments to study batch normalization, a very popular technique for training a deep convolutional network and its relationship with learning rate and batch size. In addition, the authors also propose a new initialization scheme, “ZeroInit”, to train a deep ResNet for better test accuracy. This is a very empirical study and the authors also show extensive experimental results. However, I do not see any novel findings in this study. Mostly this paper confirms the results of previous studies. The experimental results do not show much advantage of ZeroInit either. Overall, it is unclear what is the major novel contribution in this paper.
assistant
This paper conducts extensive experiments to study batch normalization, a very popular technique for training a deep convolutional network and its relationship with learning rate and batch size. In addition, the authors also propose a new initialization scheme, “ZeroInit”, to train a deep ResNet for better test accuracy. This is a very empirical study and the authors also show extensive experimental results. However, I do not see any novel findings in this study. Mostly this paper confirms the results of previous studies. The experimental results do not show much advantage of ZeroInit either. Overall, it is unclear what is the major novel contribution in this paper.