Many state of the art models rely on two architectural innovations; skip connections and batch normalization. However batch normalization has a number of limitations. It breaks the independence between training examples within a batch, performs poorly when the batch size is too small, and significantly increases the cost of computing a parameter update in some models. This work identifies two practical benefits of batch normalization. First, it improves the final test accuracy. Second, it enables efficient training with larger batches and larger learning rates. However we demonstrate that the increase in the largest stable learning rate does not explain why the final test accuracy is increased under a finite epoch budget. Furthermore, we show that the gap in test accuracy between residual networks with and without batch normalization can be dramatically reduced by improving the initialization scheme. We introduce "ZeroInit", which trains a 1000 layer deep Wide-ResNet without normalization to 94.3% test accuracy on CIFAR-10 in 200 epochs at batch size 64. This initialization scheme outperforms batch normalization when the batch size is very small, and is competitive with batch normalization for batch sizes that are not too large. We also show that ZeroInit matches the validation accuracy of batch normalization when training ResNet-50-V2 on ImageNet at batch size 1024.
This paper conducts extensive experiments to verify two practical benefits of batch normalization. i) It increases the final test accuracy and the largest stable learning rate; ii) it enables efficient training with larger batches and a larger learning rate. In addition, the authors propose a new initialization scheme, “ZeroInit”, to train a deep ResNet to improve the test accuracy. My detailed comments are as follows.
Positive points:
1. The experiments are sufficient. In this paper, the authors conduct extensive experiments to explore the benefits of batch normalization, and verifies the effectiveness of the proposed “ZeroInit”.
2. The method is effective in some cases. Specifically, the proposed “ZeroInit” outperforms batch normalization when the batch size is small, and it is competitive with batch normalization when the batch size is not too large.
Negative points:
1. The importance and novelty of the empirical study should be emphasized. The practical benefits of batch normalization can be also found in other papers. For the first benefit, most studies (Bjorck et al. 2018) have found that batch normalization is able to improve the test accuracy. For the second benefit, batch normalization requires a large batch size and a large learning rate (Santurkar et al., 2018). Therefore, what is the difference between this paper and others? More critically, it is necessary to explain why batch normalization has these benefits. It would be better to provide empirical or theoretical justifications to support these. 
2. The motivation of the proposed “ZeroInit” is not clear. (Balduzzi et al., 2017) states that “the correlations can be preserved by initializing deep networks close to linear functions”. It is not clear how “ZeroInit” preserves the correlations? 
3. Why initialize the scalar multiplier and biases to zero? What are the benefits of the zero initialization? Actually, the scalar multiplier and biases can be randomly initialized. When they are randomly initialized, what is the performance of the initialization? It is an important baseline to justify the effectiveness of the proposed initialization method. 
4. The technical details of “ZeroInit” are not clear. It would be better to express the proposed initialization “ZeroInit” in the mathematical formulation.
5. The proposed initialization “ZeroInit” is designed for deep ResNets. How to extend it to the other deep neural networks?
6. This paper states that “the empirical success of batch normalization …improves the conditioning of the loss landscape. However, our results conclusively demonstrate that this is not the case”. Does it mean that batch normalization does not improve the conditioning of the loss landscape? However, the empirical results cannot justify this statement and explain the success of batch normalization.
7. Some results of the figures are missing. In Figure 1, the experimental results of w/o batch norm with varying batch sizes (2^0 ~ 2^5) are missing. Similarly, Figure 2 also has missing results. Please provide more discussions about these missing results.