A key component of most neural network architectures is the use of normalization layers, such as Batch Normalization. Despite its common use and large utility in optimizing deep architectures, it has been challenging both to generically improve upon Batch Normalization and to understand the circumstances that lend themselves to other enhancements. In this paper, we identify four improvements to the generic form of Batch Normalization and the circumstances under which they work, yielding performance gains across all batch sizes while requiring no additional computation during training. These contributions include proposing a method for reasoning about the current example in inference normalization statistics, fixing a training vs. inference discrepancy; recognizing and validating the powerful regularization effect of Ghost Batch Normalization for small and medium batch sizes; examining the effect of weight decay regularization on the scaling and shifting parameters γ and β; and identifying a new normalization algorithm for very small batch sizes by combining the strengths of Batch and Group Normalization. We validate our results empirically on six datasets: CIFAR-100, SVHN, Caltech-256, Oxford Flowers-102, CUB-2011, and ImageNet.
The authors discuss four techniques to improve Batch Normalization, including inference example weighing, medium batch size, weight decay, the combination of batch and group normalization. Equipped with the proposed techniques, the authors obtain promising results when training deep models with various batch sizes. However, the novelty of this paper seems very limited and more experiments are required.
Please see my detailed comments below.
Positive points：
1. The proposed inference example weighing method yields promising results and does not require any re-training.
2. The combination of batch and group normalization makes it possible to train deep models with very small batch size.
3. By combining all the techniques, the proposed method yields promising performance when training deep models with different batch sizes.
Negative points:
1. Some notations are very confusing. For example, the authors use B to represent the size of a minibatch. However, why do the authors only consider B-1 samples in Eq. (2), i.e., selecting the minimum possible output among x_1, …, x_{B-1}?
2. The proposed inference example weighing method seems very similar to Batch Renormalization. Both methods seek to use a linear function to combine the batch statistics and the moving statistics. What is the essential difference between these methods?
3. What model do the authors use in the experiment of Figure 2? Why do the authors conduct experiments on different datasets in Section 3.1 (including ImageNet) and Section 3.2 (excluding ImageNet)? It would be stronger to provide ImageNet results in Section 3.2.
4. The authors draw different conclusions about the usage of weight decay from a recent work (He et al, CVPR2019). The CVPR paper reports that training \gamma and \beta without weight decay on ResNet-50 yields significant performance improvement. However, this paper shows that training ResNet-50 with weight decay improves the performance. Please comment on the differences in the conclusions.
Reference: "Bag of tricks for image classification with convolutional neural networks." CVPR, 2019.
5. The authors only report ImageNet results of the proposed inference example weighing method. However, all the experiments in Section 4 are performed on three small datasets. It is necessary and important to provide ImageNet results to show the effectiveness of the other three techniques in Section 4. 
6. Note that training deep models with non-i.i.d. minibatches is a typical case to evaluate normalization methods, e.g., Batch Renormalization. Specifically, examples in a minibatch are not sampled independently. What would happen if the authors apply the proposed techniques to the non-i.i.d. case?
7. Some closely related work should be discussed in the paper, such as
[1] "Decorrelated Batch Normalization." CVPR, 2018.
[2] "Double Forward Propagation for Memorized Batch Normalization." AAAI, 2018.
[3] "Differentiable Dynamic Normalization for Learning Deep Representation." ICML, 2019.
[4] "Iterative Normalization: Beyond Standardization towards Efficient Whitening." CVPR, 2019.
Minor issues:
1. In Section 1, the third contribution is not a complete sentence.
2. There are many typos in the paper.
(1) In Section 2, “Layer Normalization, which has found use in many natural language processing tasks.” Should “which has found use” be “which has been used”?
(2) In Section 3.1, “Batch Normalization has a disparity in function between training inference”. “between training inference” should be “between training and inference”.
(3) In Section 3.1, “we need only figure out …” should be “we only need to figure out …”