DIFFERENTIABLE LEARNING-TO-NORMALIZE VIA SWITCHABLE NORMALIZATION
We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. S-N switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been released in https://github.com/switchablenorms/.
In this work, the authors propose Switchable Normalization (SN), which *learns* to switch / select different normalization algorithms (including batch normalization (BN), Instance Normalization (IN), Layer Normalization (LN)) in layers of the networks and in different applications. The idea is motivated by observations (shown in Fig 1) that, 1) different tasks tend to have applied different normalization methods; 2) some normalization methods (e.g. BN) are fragile to very small batch size.
The authors propose a general form for different normalization methods, which is a Gaussian normalization and then scale and shift by scalars. Different normalization methods utilize different statistics as the mean and the variance of the Gaussian normalization. The authors further propose to learn the combination weights on mean and variance, which is w_k and w'_k in Eqn (3). To avoid duplicate computation, the authors also do some careful simplification on computing mean and variance with all of the three normalization methods.
In the experiment part, the authors demonstrate the effectiveness of the proposed SN method on various kinds of tasks, including ImageNet classification, object detection and instance segmentation in COCO, semantic image parsing and video recognition. In all of the tasks tested, which also cover the common application in computer vision, SN shows superior and robust performance.
Pros:
+ Neat motivation;
+ Extensive experiments;
+ Clear illustration;
Cons
- There are still some experiment results missing, as the authors themselves mentioned in the Kinetics section (but the reviewer thinks it would be ready); 
- In Page 3 the training section and Page 4, the first paragraph, it mentioned Θ and Φ (which are the weights for different normalization methods) are jointly trained and different from the previous iterative meta-learning style method. The authors attribute "In contrast, SN essentially prevents overfitting by choosing normalizers to improve both learning and generalization ability as discussed below". The reviewer does not see it is well justified and the reviewer thinks optimizing them jointly could lead to instability in the training (but it did not happen in the experiments). The authors should justify the jointly training part better.
- Page 5 the final paragraph, the reviewer does not see the point there. "We would also like to acknowledge the contributions of previous work that explored spatial region (Ren et al., 2016) and conditional normalization (Perez et al., 2017). "  Please make it a bit more clear how these works are related.