ON SELF MODULATION FOR GENERATIVE ADVER- SARIAL NETWORKS
Training Generative Adversarial Networks (GANs) is notoriously challenging. We propose and study an architectural modification, self-modulation, which improves GAN performance across different data sets, architectures, losses, regularizers, and hyperparameter settings. Intuitively, self-modulation allows the intermediate feature maps of a generator to change as a function of the input noise vector. While reminiscent of other conditioning techniques, it requires no labeled data. In a large-scale empirical study we observe a relative decrease of 5% − 35% in FID. Furthermore, all else being equal, adding this modification to the generator leads to improved performance in 124/144 (86%) of the studied settings. Selfmodulation is a simple architectural change that requires no additional parameter tuning, which suggests that it can be applied readily to any GAN.
The paper examines an architectural feature in GAN generators -- self-modulation -- and presents empirical evidence supporting the claim that it helps improve modeling performance. The self-modulation mechanism itself is implemented via FiLM layers applied to all convolutional blocks in the generator and whose scaling and shifting parameters are predicted as a function of the noise vector z. Performance is measured in terms of Fréchet Inception Distance (FID) for models trained with and without self-modulation on a fairly comprehensive range of model architectures (DCGAN-based, ResNet-based), discriminator regularization techniques (gradient penalty, spectral normalization), and datasets (CIFAR10, CelebA-HQ, LSUN-Bedroom, ImageNet). The takeaway is that self-modulation is an architectural feature that helps improve modeling performance by a significant margin in most settings. An ablation study is also performed on the location where self-modulation is applied, showing that it is beneficial across all locations but has more impact towards the later layers of the generator.
I am overall positive about the paper: the proposed idea is simple, but is well-explained and backed by rigorous evaluation. Here are the questions I would like the authors to discuss further:
- The proposed approach is a fairly specific form of self-modulation. In general, I think of self-modulation as a way for the network to interact with itself, which can be a local interaction, like for squeeze-and-excitation blocks. In the case of this paper, the self-interaction allows the noise vector z to interact with various intermediate features across the generation process, which for me appears to be different than allowing intermediate features to interact with themselves. This form of noise injection at various levels of the generator is also close in spirit to what BigGAN employs, except that in the case of BigGAN different parts of the noise vector are used to influence different parts of the generator. Can you clarify how you view the relationship between the approaches mentioned above?
- It’s interesting to me that the ResNet architecture performs better with self-modulation in all settings, considering that one possible explanation for why self-modulation is helpful is that it allows the “information” contained in the noise vector to better propagate to and influence different parts of the generator. ResNets also have this ability to “propagate” the noise signal more easily, but it appears that having a self-modulation mechanism on top of that is still beneficial. I’m curious to hear the authors’ thoughts in this.
- Reading Figure 2b, one could be tempted to draw a correlation between the complexity of the dataset and the gains achieved by self-modulation over the baseline (e.g., Bedroom shows less difference between the two approaches than ImageNet). Do the authors agree with that?