WHITENING AND COLORING BATCH TRANSFORM FOR GANS
Batch Normalization (BN) is a common technique used to speed-up and stabilize training. On the other hand, the learnable parameters of BN are commonly used in conditional Generative Adversarial Networks (cGANs) for representing classspecific information using conditional Batch Normalization (cBN). In this paper we propose to generalize both BN and cBN using a Whitening and Coloring based batch normalization. We show that our conditional Coloring can represent categorical conditioning information which largely helps the cGAN qualitative results. Moreover, we show that full-feature whitening is important in a general GAN scenario in which the training process is known to be highly unstable. We test our approach on different datasets and using different GAN networks and training protocols, showing a consistent improvement in all the tested frameworks. Our CIFAR-10 conditioned results are higher than all previous works on this dataset.
This paper proposed Whitening and Coloring (WC) transform to replace batch normalization (BN) in generators for GAN. WC generalize BN by normalizing features with decorrelating (whitening) matrix, and then denormalizing (coloring) features by learnable weights. The main advantage of WC is that it exploits the full correlation matrix of features, while BN only considers the diagonal. WC is differentiable and is only 1.32x slower than BN. The authors also apply conditional WC, which learn the parameters of coloring conditioned on labels, to conditional image generation.  Experimental results show WC achieves better inception score and FI distance comparing to BN on CIFAR-10, CIFAR-100, STL-10 and Tiny Imagenet. Furthermore, the conditional image generation results by WC are better than all previous methods.
I have some detailed comments below.
+ The paper is well written, and I generally enjoyed reading the paper.
+ The experimental results look sufficient, and I appreciate the ablation study sections. 
+ The score on supervised CIFAR-10 is better than previous methods. 
- The main text is longer than expectation. I would suggest shorten section 3.1 Cholesky decomposition, section 4 conditional color transformation and the text in section 5 experiments.
- The proposed WC transform is general. It is a bit unclear why it is particularly effective for generator in GAN. Exploiting the full correlation matrix sounds reasonable, but it may also introduce unstability. It would help if the authors have an intuitive way to show that whitening is better than normalization.
- It is unclear why conditional WC can be used for generation conditioned on class labels. In Dumoulin 2016, conditional instance normalization is used for generating images conditioned on styles. As image styles are described by Gram matrix (correlation) of features, changing first order and second order statistics of features is reasonable for image generation conditioned on styles. I cannot understand why conditional WC can be used for generation conditioned on class labels. I would like the authors to carefully explain the motivation, and also provide visual results like using the same random noise as input, but only changing the class conditions. 
- It is unclear to me why the proposed whitening based on Cholesky decomposition is better than ZCA-based in Huang 2018. Specifically, could the authors explain why WC is better than W_{aca}C in Table 3? 
- The authors claim progressive GAN used a larger generator to achieve a better performance than WC. The WC layer is generally larger than BN layer and has more learnable parameters. Could the authors compare  the number of parameter of generator in BN-ResNet, WC-ResNet, and progressive GAN?
- In Table 3, std-C is better than WC-diag, which indicates coloring is more important. In Table 6, cWC-diag is better than c-std-C, which indicates whitening is more important. Why? 
- What is the batch size used for training? For conditional WC, do the samples in each minibatch have same label?
- Having ImageNet results will be a big support for the paper.
===========  comments after reading rebuttal ===========
I appreciate the authors' feedback. I raised my score for Fig 7 showing the conditional images, and for experiments on ImageNet. 
I think WC is a reasonable extension to BN, and I generally like the extensive experiments. However, the paper is still borderline to me for the following concerns.
- I strongly encourage the authors to shorten the paper to the recommended 8-page. 
- The motivation of WC for GAN is still unclear. WC is general extension of BN, and a simplified version has been shown to be effective for discrimination in Huang 2018. I understand the empirically good performance for GAN. But I am not convinced why WC is particularly effective for GAN, comparing to discrimination. The smoothness explanation of BN applies to both GAN and discrimination. I actually think it may be nontrivial to extend the smoothness argument from BN to WC.
- The motivation of cWC is still unclear. I did not find the details of cBN for class-label conditions, and how they motivated it in (Gulrajani et al. (2017) and (Miyato et al. 2018). Even if it has been used before, I would encourage the authors to restate the motivation in the paper. Saying it has been used before is an unsatisfactory answer for an unintuitive setting.
- Another less important comment is that it is still hard to say how much benefits we get from the more learnable parameters in WC than BN. It is probably not so important because it can be a good trade-off for state-of-the-art results. In table 3 for unconditioned generation, it looks like the benefits come a lot from the larger parameter space. For conditioned generation in table 6, I am not sure if whitening is conditioned or not, which makes it less reliable to me. If whitening is conditioned, then the samples in each minibatches may not be enough to get a stable whitening. If whitening is unconditioned, then there seems to be a mismatch between whitening and coloring. 
====== second round after rebuttal =============
I raise the score again for the commitment of shortening the paper and the detailed response from the authors. That being said, I am not fully convinced about motivations for WC and cWC. 
- GAN training is more difficult and unstable, but that does not explain why WC is particularly effective for GAN training. 
- I have never seen papers saying cBN/cWC is better than other conditional generator conditioned on class labels. I think the capacity argument is interesting, but I am not sure if it applies to convolutional net (where the mean and variance of a channel is used), or how well it can explain the performance because neural nets are overparameterized in general. I would encourage authors to include these discussions in the paper.