A well-known issue of Batch Normalization is its significantly reduced effectiveness in the case of small mini-batch sizes. When a mini-batch contains few examples, the statistics upon which the normalization is defined cannot be reliably estimated from it during a training iteration. To address this problem, we present Cross-Iteration Batch Normalization (CBN), in which examples from multiple recent iterations are jointly utilized to enhance estimation quality. A challenge of computing statistics over multiple iterations is that the network activations from different iterations are not comparable to each other due to changes in network weights. We thus compensate for the network weight changes via a proposed technique based on Taylor polynomials, so that the statistics can be accurately estimated and batch normalization can be effectively applied. On object detection and image classification with small mini-batch sizes, CBN is found to outperform the original batch normalization and a direct calculation of statistics over previous iterations without the proposed compensation technique.
Paper summary: This paper proposes a new normalization technique specially designed for settings with small mini-batch sizes (where previous methods like BatchNorm are known to suffer). The approach aggregates mean/variance statistics from previous iterations, weighted based on the Taylor expansion, to get a better estimate of population statistics. The authors evaluate their approach on ImageNet classification, and object detection and instance segmentation on the COCO dataset.
The paper is clearly written and explores an interesting idea---aggregating mini-batch statistics across iterations. That being said, I am not convinced by the utility of the proposed approach since it doesn’t offer over significant benefits over prior approaches designed for the small mini-batch setting (e.g., Group Normalization)---either in terms of empirical performance, implementation complexity, or lower computational/memory requirements.
Specific comments/questions:
1. Why do the authors not include the Kalman Normalization baseline in the paper? Based on my understanding, it is also designed for the low-sample regime (and the original paper also conducts experiments on ImageNet/COCO). Also the BRN baseline is included in Table 3 for ImageNet but is missing from the COCO experiments. It is important to thoroughly compare to other normalization techniques specifically designed for this regime to clearly highlight relative benefits of the proposed approach.
2. The authors mention that they average performance across 5 trials but omit confidence intervals in their Figures/Tables. Since the difference in performance between the different approaches compared is small, I think confidence intervals should be reported to see whether improvements are within statistical error margins. In fact, for CIFAR-10 based on Table 7 in the Appendix, performance of batch renormalization, group normalization and CBN is essentially the same.
3. In Table 3, how many iterations was the BRN aggregation performed over? Was it also chosen to ensure effective number of samples was not less than 16 (like CBN). What do Figures 3 and 4 look like for BRN?
4. One thing I find interesting (the authors do not discuss this specifically) about Figure 5 in the Appendix is that both for CIFAR and COCO (the two datasets for which plots are provided), CBN helps as long as it is used before the final learning rate drop. Specifically, choosing a burn-in period as large as 120 for CIFAR and 8 for COCO is fine (in fact, probably the best) as long as you turn on CBN before the final learning rate drop. This makes me curious whether BN in the low-sample regime only suffers in terms of generalization performance in the final stages of training (compared to low-sample alternatives like GN/CBN, etc).
The authors should include other recent benchmarks (Kalman normalization and BRN in the omitted Tables) and error bars to make the change in performance clearer. It would also be interesting to see whether (and probably make the paper stronger if) alternatives like GN/KN benefit from being combined with the proposed scheme to aggregate statistics over time.
Overall, while the proposed approach is novel, its performance is comparable to prior approaches, with the disadvantage of an additional computational/memory footprint. Thus, I am not yet convinced about how useful/interesting this approach would be to the community.