In this paper, we propose a generalization of the BN algorithm, diminishing batch normalization (DBN), where we update the BN parameters in a diminishing moving average way. Batch normalization (BN) is very effective in accelerating the convergence of a neural network training phase that it has become a common practice. Our proposed DBN algorithm remains the overall structure of the original BN algorithm while introduces a weighted averaging update to some trainable parameters. We provide an analysis of the convergence of the DBN algorithm that converges to a stationary point with respect to trainable parameters. Our analysis can be easily generalized for original BN algorithm by setting some parameters to constant. To the best knowledge of authors, this analysis is the first of its kind for convergence with Batch Normalization introduced. We analyze a two-layer model with arbitrary activation function. The primary challenge of the analysis is the fact that some parameters are updated by gradient while others are not. The convergence analysis applies to any activation function that satisfies our common assumptions. For the analysis, we also show the sufficient and necessary conditions for the stepsizes and diminishing weights to ensure the convergence. In the numerical experiments, we use more complex models with more layers and ReLU activation. We observe that DBN outperforms the original BN algorithm on Imagenet, MNIST, NI and CIFAR-10 datasets with reasonable complex FNN and CNN models.
In this work, the authors propose a generalization of the batch normalization (BN) technique often used in training neural networks, and analyzed this convergence. In particular, a one hidden layer and one BN hidden layer fully connected network is considered, and a deterministic gradient descent algorithm with certain kind of BN has been considered in this work. The proposed “generalized” BN strategy is devised on the deterministic setting, but it is a slight generalization of the original BN by introducing a moving average operation. Classical results of Bertsekas is leveraged to show the asymptotic convergence of the algorithm. 
I have the following three main comments about the paper. 
1)	Only deterministic setting is considered, but in this case every time the entire data set will be used to perform the averaging, it appears to be much easier to analyze than the stochastic setting. Further the reviewer has doubt on whether the resulting deterministic algorithm has any practice value. 
2)	Because the authors have used the Bertsekas/Tsitsiklis (B/T) argument, only asymptotic convergence is shown. It is not clear, even in the deterministic case, whether some kind of sublinear convergence rate can be obtained. 
3)	Only one hidden layer of neural network with one BN operation is considered. It is not clear whether the analysis can be extended to multiple layers, despite the statement of the author saying that “the technique presented can be extended to more layers with additional notation”. In particular, when there are multiple layers, the BN layers will be further composite together across multiple nonlinear operations. 
4)	The authors have mentioned that the derivative is always taking w.r.t. theta. However, in (9) is appears that the derivative is taken with respect to lambda, in order to get the Lipschitz condition on \lambda. This is a bit confusing. Also it is not clear how the gradient in Assumption 5 is defined. 
5)	Assumption 5 does not make sense. Problem (1) is a constrained problem with both variables being confined in compact feasible sets. And this condition is important in Assumption2. Now the authors say that at stationary solution the gradient is zero? Please specify functions when this will happen. I will suggest that the authors use a proper definition of stationarity solution for constrained problems. 
6)	Follow up on the previous point. The analysis builds upon B/T argument for unconstrained optimization. However it is not suitable for the constrained problems that the authors started out at the beginning of the paper. The authors may consider develop new analysis tools to understand the problem at hand, rather than assuming away the difficulties.