Generative adversarial networks (GANs) have shown great success in applications such as image generation and inpainting. However, they typically require large datasets, which are often not available, especially in the context of prediction tasks such as image segmentation that require labels. Therefore, methods such as the CycleGAN use more easily available unlabelled data, but do not offer a way to leverage additional labelled data for improved performance. To address this shortcoming, we show how to factorise the joint data distribution into a set of lower-dimensional distributions along with their dependencies. This allows splitting the discriminator in a GAN into multiple "sub-discriminators" that can be independently trained from incomplete observations. Their outputs can be combined to estimate the density ratio between the joint real and the generator distribution, which enables training generators as in the original GAN framework. We apply our method to image generation, image segmentation and audio source separation, and obtain improved performance over a standard GAN when additional incomplete training examples are available. For the Cityscapes segmentation task in particular, our method also improves accuracy by an absolute 14.9% over CycleGAN while using only 25 additional paired examples.
I found this paper very easy and clear to follow - the authors present, what I believe to be an elegant, approach to training a GAN in the presence of missing data or where many marginal samples might be available but very few complete (e.g. paired) samples. The approach proceeds by identifying that the joint distributions (true and approximate) can be factored so as to yield a number of different density ratios which can then be estimated by specific discriminators; in particular, these include the appropriate marginal density ratios and then corresponding overall correction factors. As a caveat to the review I should point out that while I am familiar with GANs, they are not my main area of expertise so this should be taken into consideration - apologies if there is literature I have missed.
Experiments: The authors provide a number of illustrative experiments that demonstrate the efficacy of the approach across a number of tasks. There are many differing GAN models but due to the nature of the problem I don't have a big issue with the majority of the comparisons being against a standard GAN since the tasks are suitably designed. For the paired MNIST experiment I found it hard to assess the qualitative results visually and am always concerned about the ad-hoc nature of Inception Distances - I find it difficult to attribute weight to them quantitatively since they are usually being used to assess things where they might suffer from a common error (e.g. they are both based on NNs). Also, I'm not fully on board with the dependency metric in (5) but then the authors also point out the same concerns. The other experiments I found more convincing.
I appreciated having error bars on some of the plots to help assess significance - would it not be possible to put error bars on all plots?
I found the additional extensions presented in the appendix to be interesting ideas as well and would be interested to see how the approach works with other GAN objectives as mentioned for future work.
I am mostly very positive about this work - my main concern is really common to most GANs - all the analysis relies on the premise that the discriminators can be setup as good estimators for the density ratios. We know that this is not always the case since everything comes from samples and if the capacities of each of the discriminators are not set appropriately then I would expect problems to occur - has this been explored by the authors? It would be no detriment to the work to include failure examples where the authors purposefully make use of inappropriate architectures for some of the discriminators to check for this? For example, there will be large imbalances in the number of training samples used for the different discriminators - how does this affect stability?
Other notes:
- Whilst I understand the point about independent marginals in 2.4 I'm not sure I see the motivation as clearly since it seems that the model is much more useful when there is dependent information but maybe there's a use-case I'm not thinking of?