Kalman Normalization: Normalizing Internal Representations Across Network Layers
As an indispensable component, Batch Normalization (BN) has successfully improved the training of deep neural networks (DNNs) with mini-batches, by normalizing the distribution of the internal representation for each hidden layer. However, the effectiveness of BN would diminish with the scenario of micro-batch (e.g. less than 4 samples in a mini-batch), since the estimated statistics in a mini-batch are not reliable with insufficient samples. This limits BN's room in training larger models on segmentation, detection, and video-related problems, which require small batches constrained by memory consumption. In this paper, we present a novel normalization method, called Kalman Normalization (KN), for improving and accelerating the training of DNNs, particularly under the context of microbatches. Specifically, unlike the existing solutions treating each hidden layer as an isolated system, KN treats all the layers in a network as a whole system, and estimates the statistics of a certain layer by considering the distributions of all its preceding layers, mimicking the merits of Kalman Filtering. On ResNet50 trained in ImageNet, KN has 3.4% lower error than its BN counterpart when using a batch size of 4; Even when using typical batch sizes, KN still maintains an advantage over BN while other BN variants suffer a performance degradation. Moreover, KN can be naturally generalized to many existing normalization variants to obtain gains, e.g.equipping Group Normalization [34] with Group Kalman Normalization (GKN). KN can outperform BN and its variants for large scale object detection and segmentation task in COCO 2017.
This paper addresses the batch normalization problem with small batch size. Traditional batch normalization relies on the estimation of the batch statistics (mean and variance) over each batch. Batch normalization requires relatively large batch size in order to obtain relatively reliable estimates of these statistics. However, due to memory constraint, some higher-level tasks (e.g., in computer vision) could not use large batchsize. Therefore, developing normalization method with small batch size is important for improving the performance of these systems. Experient results show that the proposed KN outperform previous batch normalization approaches both in regular batch sizes and small batch sizes.  The problem formulation and the proposed algorithm have a gap from the canonical Kalman filter and it is not discussed clearly in the paper. It is a bit confusing about the formulation of the Kalman filter. What is the state x^k in (2)? Is it the activation vector at the k-th hidden layer or the groundtruth mean at the hidden layer? I think a correct formulation of Kalman filter for this problem is to model the state as the true mean value if the observation is modeled to be the (noisy) observation of this mean and u^k is the disturbance due to inaccurate modeling with a linear dynamic system. Otherwise, if the state is modeled to be the realization of the hidden activation vector at the k-th layer, then \hat{\mu}_{k|k}, which in Kalman filter context is the conditional mean of the state given all the observations up to k, is an minimum-mean-square-error (MMSE) estimate of the realization of the activation vector. In batch normalization problem, the desired quantity to estimate should be the mean not the realization of the activation vector. So the current formulation in (2) is not correct. Furthermore, the proposed algorithm is quite different from standard Kalman filter, where the gain matrix (instead of a gain scalar q^k) is computed from a Riccati equation (a nonlinear recursion over the covariance matrix). None of these are sufficiently discussed.