Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks
Real-world image recognition is often challenged by the variability of visual styles including object textures, lighting conditions, filter effects, etc. Although these variations have been deemed to be implicitly handled by more training data and deeper networks, recent advances in image style transfer suggest that it is also possible to explicitly manipulate the style information. Extending this idea to general visual recognition problems, we present Batch-Instance Normalization (BIN) to explicitly normalize unnecessary styles from images. Considering certain style features play an essential role in discriminative tasks, BIN learns to selectively normalize only disturbing styles while preserving useful styles. The proposed normalization module is easily incorporated into existing network architectures such as Residual Networks, and surprisingly improves the recognition performance in various scenarios. Furthermore, experiments verify that BIN effectively adapts to completely different tasks like object classification and style transfer, by controlling the tradeoff between preserving and removing style variations. BIN can be implemented with only a few lines of code using popular deep learning frameworks.
Update following the author rebuttal:  I would like to thank the authors for providing a thoughtful rebuttal and addressing the points I raised in my review.  -----  The paper proposes a method -- termed Batch-Instance Normalization (BIN) -- for automatically learning which of batch (BN) or instance (IN) normalization should be employed for individual feature maps in a convolutional neural network. The justification is simple and intuitive: in certain cases it is beneficial to standardize features on a per-example basis (e.g., in most cases global illumination is irrelevant for object classification), but in other cases doing so is hurtful (e.g., when predicting weather or time global illumination is a very informative feature).  The authors propose a simple idea: batch normalization and instance normalization are blended through a convex combination of both forms of normalization, and the blending coefficient is treated as a model parameter and learned jointly with the rest of the network parameters. I like the simplicity of the proposed method. The paper is well-written, straightforward, and contextualizes the idea well alongside relevant work.  The authors present experimental results for object classification, multi-domain training, and artistic style transfer. For object classification, accuracies and learning curves are presented for ResNet architectures trained on CIFAR10, CIFAR100, and ImageNet, showing that BIN consistently outperforms BN, IN, and BN+IN (i.e., BIN with a constant blending coefficient of 0.5) by a small margin. In particular, the comparison with BN+IN suggests that the adaptive nature of the proposed idea is responsible for the observed improvements, rather than the the ensembling of BN and IN. The authors also analyze the distribution of blending coefficients and show that the distribution is roughly bimodal, with most of the probability mass concentrated near the BN end of the spectrum and a smaller amount concentrated near the IN end of the spectrum, which aligns well with the intuition put forward by the paper. One criticism I have to voice is that the absence of confidence intervals for the presented accuracies makes it hard for me to evaluate whether the improvements presented are statistically significant in some cases. The consistent nature of the improvements -- best exemplified by Table 2 -- goes some way towards building confidence in their statistical significance, but I would nevertheless like to have a sense of what sort of variability to expect due to simple factors like random weight initialization and whether the observed improvements deviate significantly from it.  For multi-domain training, the authors compare BIN to BN in a mixed-domain classification setting. Once again, consistent improvements are shown, and the authors conclude that BIN’s ability to alleviate style disparities between domains is responsible for the increase in accuracies. I believe this is a likely explanation, but I would like to see whether BIN also improves performance when training on individual domains, as it could be that the ability to alleviate intra-domain style disparities is also helpful. The authors also compare BIN to BN in a domain-adaptation setting and show that BIN outperforms BN in most settings by a small margin.  For style transfer, the paper compares BIN with BN, IN, and BN+IN when training a feed-forward style transfer network on a single style image. They conclude that BIN makes a trade-off between the content loss and the style loss (which are found to be respectively slightly lower and slightly higher than their BN counterparts), and that BIN is better than BN+IN at minimizing both the content and style losses. Presenting results on more than one style would help build confidence in the significance of the observed differences; alternatively, presenting results for a multi-style feedforward network like the ones proposed by Dumoulin et al., Huang et al., or Ghiasi et al. (G. Ghiasi, H. Lee, M. Kudlur, V. Dumoulin, J. Shlens. “Exploring the structure of a real-time, arbitrary neural artistic stylization network”. Proceedings of the British Machine Vision Conference (2017).) would be equally convincing.  In summary, the paper proposes a simple idea, explains it in a straightforward and well-written fashion, and presents thorough and convincing empirical evidence that it improves performance in multiple problem settings. The biggest criticism I have to voice is that I would like to see confidence intervals for the accuracies. Although the idea itself is not groundbreaking, I believe it is a useful and relevant addition to a deep learning researcher’s toolkit.