Most existing GANs architectures that generate images use transposed convolution or resize-convolution as their upsampling algorithm from lower to higher resolution feature maps in the generator. We argue that this kind of fixed operation is problematic for GANs to model objects that have very different visual appearances. We propose a novel adaptive convolution method that learns the upsampling algorithm based on the local context at each location to address this problem. We modify a baseline GANs architecture by replacing normal convolutions with adaptive convolutions in the generator. Experiments on CIFAR-10 dataset show that our modified models improve the baseline model by a large margin. Furthermore, our models achieve state-of-the-art performance on CIFAR-10 and STL-10 datasets in the unsupervised setting.
The paper operates under the hypothesis that the rigidity of the convolution operator is responsible in part for the poor performance of GANs on diverse visual datasets. The authors propose to replace convolutions in the generator with an Adaptive Convolution Block, which learns to generate the convolution weights and biases of the upsampling operation adaptively for each pixel location. State-of-the-art Inception scores are presented for the CIFAR-10 and STL-10 datasets.
I think the idea of leveraging adaptive convolutions in decoder-based models is compelling, especially given its success in video frame interpolation, which makes me wonder why the authors chose to restrict themselves to GANs. Wouldn't the arguments used to justify replacing regular convolutions in the generator with adaptive convolution blocks apply equally well to any other decoder-based generative model, like a VAE, for instance?
I find the paper lacking on the evaluation front. The evaluation of GANs is still very much an open research problem, which means that making a compelling case for the effectiveness of a proposed method requires nuance and contextualization. The authors claim a state-of-the-art Inception score but fail to explain what argument this claim supports. This is important, because the Inception score is not a universal measure of GAN performance: it provides a specific view on the ability of a generator to cover human-defined modes in the data distribution, but it does not inform on intra-class mode coverage and is blind to things like the generator collapsing on one or a few template samples per class.
I am also surprised that the relationship with HyperNetworks [1] is not outlined, given that both papers leverage the idea of factoring network parameters through a second neural network.
Some additional comments:
- Figure 1 should be placed much earlier in the paper, preferably above Section 3. In its current state, the paper provides a lot of mathematical notation to digest without any visual support.
- "[...] a transposed convolution is equivalent to a convolution [...]": This is inaccurate. A convolution's backward pass is a transposed convolution and vice versa, but they are not equivalent (especially when non-unit strides are involved).
- "The difficulties of training GANs is well known": There is a grammatical error in this sentence.
- "If [the discriminator] is too strong, log(1 - D(G(z))) will be close to zero and there would be almost no gradient [...]": This is only true for the minimax GAN objective, which is almost never used in practice. The non-saturating GAN objective does not exhibit this issue, as [2] re-iterated recently.
- "Several works have been done [...]": There is a grammatical error here.
- The WGAN-GP citation is wrong (Danihelka et al. rather than Gulrajani et al.).
Overall, the paper's lack of sufficient convincing empirical support prevents me from recommending its acceptance.
References:
[1] Ha, D., Dai, A., and Le, Q. V. (2016). HyperNetworks. arXiv:1609.09106.
[2] Fedus, W., Rosca, M., Lakshminarayanan, B., Dai, A. M., Mohamed, S., and Goodfellow, I. (2017). Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step. arXiv:1710.08446.