Batch Normalization (BN) is a well-known technique used in training deep neural networks. The main idea behind batch normalization is to normalize the features of the layers (i.e., transforming them to have a mean equal to zero and a variance equal to one). Such a procedure encourages the optimization landscape of the loss function to be smoother, and improve the learning of the networks for both speed and performance. In this paper, we demonstrate that the performance of the network can be improved, if the distributions of the features of the output in the same layer are similar. As normalizing based on mean and variance does not necessarily make the features to have the same distribution, we propose a new normalization scheme: Batch Normalization with Skewness Reduction (BNSR). Comparing with other normalization approaches, BNSR transforms not just only the mean and variance, but also the skewness of the data. By tackling this property of a distribution, we are able to make the output distributions of the layers to be further similar. The nonlinearity of BNSR may further improve the expressiveness of the underlying network. Comparisons with other normalization schemes are tested on the CIFAR-100 and ImageNet datasets. Experimental results show that the proposed approach can outperform other state-of-the-arts that are not equipped with BNSR.
The paper proposes to add an extra nonlinearity function in batch normalization, between the normalization and affine scaling. The nonlinearity is a power function x ** p for x >= 0 and - (-x) ** p for x < 0 (python pseudo-code), where p is a constant, and the authors propose to keep it fixed to 1.01. The intuition behind is reducing skewness of activations, and the modification is evaluated on CIFAR and tiny ImageNet datasets. The authors also experiment with the part at which to insert the nonlinearity.
I propose reject mainly due to insufficient experimental evaluation. The authors choose small datasets on which trained networks have large variance, and report a single accuracy value for each network, so it is not possible to judge the effectiveness of the method. Regarding the skewness reduction itself, it is not very convincing too, because the authors use a very small value for p (1.01), and because after affine layer ReLU removes most of the negative part, so perhaps the negative part of the nonlinearity is not needed.
I would suggest the following experiments to improve the paper:
 - try various values of p
 - try removing negative part of the nonlinearity
 - report mean+-std results of 5 runs for each experiment on small dataset
 - include experiments on full scale ImageNet
 - include more network architectures, or at least have more configurations of ResNet.
Would be very helpful if authors included code in pytorch or tensorflow for the proposed modification. It is surprising that such a small addition increases epoch time from 86 s to 119 s.
Also, there are some minor grammar mistakes. There is an undefined figure reference on page 6, and it is not clear what figures 6 and 7 are supposed to show, since the colors of histograms are never explained.