Neural networks with ReLU activation functions have demonstrated their success in many applications. Recently, researchers noticed a potential issue with the optimization of ReLU networks: the ReLU activation functions are positively scaleinvariant (PSI), while the weights are not. This mismatch may lead to undesirable behaviors in the optimization process. Hence, some new algorithms that conduct optimizations directly in the path space (the path space is proven to be PSI) were developed, such as Stochastic Gradient Descent (SGD) in the path space, and it was shown that, SGD in the path space is superior to that in the weight space. However, it is still unknown that whether other deep learning techniques beyond SGD, such as batch normalization (BN), could also have their counterparts in the path space. In this paper, we conduct a formal study on the design of BN in the path space. According to our study, the key challenge is how to ensure the forward propagation in the path space, because BN is utilized during the forward process. To tackle such challenge, we propose a novel re-parameterization of ReLU networks, with which we replace each weight in the original neural network, with a new value calculated from one or several paths, while keeping the outputs of the network unchanged for any input. Then we show that BN in the path space, namely P-BN, is just a slightly modified conventional BN on the re-parameterized ReLU networks. Finally, our experiments on two benchmark datasets, CIFAR and ImageNet, show that the proposed P-BN can significantly outperform the conventional BN in the weight space.
This paper analyzes a reparametrization of the network that migrates from the weight space to the path space. It enables an easier way to understand the batch normalization (BN). Then the authors propose a variant of BN on the path space and empirically show better performance with the new proposal. 
To study BN in the reparameterized space is well-intuited and a natural idea. Theorem 3.1 itself is interesting and has some value in understanding BN. However, the main contribution of the paper, i.e., the proposal of the P-BN, is not motivated enough. It is merely mentioned in the beginning of section 4 and it's not clear why this modification is better compared to conventional BN. This is not verified by theory either. By comparing theorem 3.2 and theorem 4.1, it seems P-BN even gives even worse upper bound of gradient norm. 
Plus we don't actually care that much about the issue of gradient exploding since one could always do gradient clipping. The notorious gradient vanishing problem on the other hand, is not address in the theorems. 
The formulation of the P-BN seems to be closely related to ResNet, since it sets aside the identity mapping and only normalizes on the other part. It would be better to have some discussions. 
Also, the reparameterization and P-BN seems only to apply to fully connected layer from Eqn. (3-5) where they are proposed, but the experiments applies to ResNet. It would be better to describe the method in a broader sense. How would you do this P-BN in more complicated networks?
Finally, it's very unclear to me the value of Theorem 3.1 and the proof that takes almost one page in the main context. The assumption of diagonal elements of matrix w to be all positive is very restrictive and simply removes the effect of ReLU activations. 
Therefore I think the paper has some room for improvement and is not very suitable for publication right now.