Deep Neural Networks (DNNs) thrive in recent years in which Batch Normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the reduction operations. In this paper, we propose alleviating the BN's cost by using only a small fraction of data for mean & variance estimation at each iteration. The key challenge to reach this goal is how to achieve a satisfactory balance between normalization effectiveness and execution efficiency. We identify that the effectiveness expects less data correlation while the efficiency expects regular execution pattern. To this end, we propose two categories of approach: sampling or creating few uncorrelated data for statistics' estimation with certain strategy constraints. The former includes "Batch Sampling (BS)" that randomly selects few samples from each batch and "Feature Sampling (FS)" that randomly selects a small patch from each feature map of all samples, and the latter is "Virtual Dataset Normalization (VDN)" that generates few synthetic random samples. Accordingly, multi-way strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. All the proposed methods are comprehensively evaluated on various DNN models, where an overall training speedup by up to 21.7% on modern GPUs can be practically achieved without the support of any specialized libraries, and the loss of model accuracy and convergence rate are negligible. Furthermore, our methods demonstrate powerful performance when solving the well-known "micro-batch normalization" problem in the case of tiny batch size.
This paper proposes a new technique that can reduce the computational complexity of batch normalization. Several sampling methods called NS, BS, and FS are proposed, and additionally, VDN is proposed to generate random virtual samples.  Experiment results follow to support the authors' goal.
pros)
(+) The paper is clearly written and easy to follow.
(+) The way of reducing the computational cost looks good.
(+) The method can be easily adapted to BN or other batch-based methods.
cons)
(-) Any motivations or insights into NS, BS, and FS are not provided. Furthermore, the proposed sampling strategy looks heuristic without any studies.
(-) For VDN, how to generate virtual samples is not clearly stated. I think the way of generating samples is critical to the performance of VDN but hard to find the exact way to do that.
(-) How to determine the sampling ratio for each normalization method is not provided, and it would be better if the authors can show some studies about sampling ratio versus the speed gain.
(-) It is hard to choose which normalization among FS and BS is better as looking at Table 2 and 3 only. So how about the speedup using BS+VDN?  
comments)
- It is something strange why the authors used shallower ResNet on ImageNet and deeper ones on CIFAR datasets, maybe it was due to the training time, but the authors should clarify it.
- What is the goal of the correlation analysis section? Especially, Figure 7 looks similar among BS, FS, VDN, and NS. Furthermore, the authors could include BN into the comparison.
- This kind of paper should incoporate different ablation studies as much as the authors can, but it seems to be lacking.
The paper has an interesting idea about sampling some features to speed up the batch normalization. However, it looks quite obvious and needs more experimental grounds such as ablation studies to support the idea.