Semi-supervised Learning with GANs: Manifold Invariance with Improved Inference
Semi-supervised learning methods using Generative adversarial networks (GANs) have shown promising empirical success recently. Most of these methods use a shared discriminator/classifier which discriminates real examples from fake while also predicting the class label. Motivated by the ability of the GANs generator to capture the data manifold well, we propose to estimate the tangent space to the data manifold using GANs and employ it to inject invariances into the classifier. In the process, we propose enhancements over existing methods for learning the inverse mapping (i.e., the encoder) which greatly improves in terms of semantic similarity of the reconstructed sample with the input sample. We observe considerable empirical gains in semi-supervised learning over baselines, particularly in the cases when the number of labeled examples is low. We also provide insights into how fake examples influence the semi-supervised learning procedure.
Chiefly theoretical work with some empirical results on SVHN and CIFAR10. This paper proposes using a trained GAN to estimate mappings from and to the true data distribution around a data point, and use a kind of neural PCA to estimate tangents to those estimates, then used for training a manifold-invariant classifier. Some additional work investigating regular GAN vs feature matching GAN is briefly presented, and an augmentation to BiGAN.
It feels a bit like this is two works squeezed into one. Maybe three.
There is an exploration of FM-GAN vs a regular GAN training objective, with some nice results shown where the classification entropy (confidence of the classification, as I read it) is much better for an FM-GAN than for a regular GAN.
There is an Augmented BiGAN which achieves nicely lower classification error than BiGAN on infer-latents-then-generate g(h(x)).
The most substantial work presented here is the manifold-invariance. The first thing I wrestle with is that the method is a bit complex, making it probably tricky for others to get right, and complex/fiddly to implement. In particular, 2.1.2 proposes to freeze f, g, and h, and introduce p and pbar as a nonlinear approximation to SVD. This introduces the second thing I wrestle with: numerous layered approximations. The method requires g and h to be good approximations to generate and infer to and from the data manifold. The results (e.g. figure 2) do not indicate that these approximations are always very good. The method requires that p and pbar reasonably capture singular directions in the latent space, but figure 3 shows this approximation only sort-of holds. This has me wondering about transferability, and wondering how to measure each of the approximation errors to evaluate whether the method is useful for a dataset. The CIFAR-10 results reinforce my concern.
The MNIST result in line 264 (0.86) is quite good for semi-supervised. What is the difference between the results in 264-265 vs the results in Table 1? Different numbers are given for SVHN in each location, yet line 251 suggests the results in both locations are semi-sup. Work would feel more complete with comparisons on semi-sup MNIST in Table 1, then you could show ladder, CAE, MTC, etc. I'm guessing you're up against space constraints here...
Table 1 missing some competitive results. A quick Google search for svhn semi supervised gives https://arxiv.org/abs/1610.02242 showing 7.05% for SVHN with 500 labels, 5.43% with 1000 labels; 16.55% CIFAR10@4k. https://papers.nips.cc/paper/6333-regularization-with-stochastic-transformations-and-perturbations-for-deep-semi-supervised-learning.pdf reports 6.03% on SVHN with 1% of labels (~700).
Missing conclusion/future directions.
Minor nitpicks:
grammar of lines 8-9 needs work
grammar of lines 42-43
lines 72/73 have two citations in a row
lines 88/89 unnecessary line break
line 152 wrong R symbol
line 204 comma before however
lne 295 'do not get as better' grammar needs work
All in all, the results are OK-to-good but not universally winning. The topic is of interest to the community, including a few novel ideas. (The paper perhaps should be multiple works.) It looks like a new SOTA is presented for SVHN semisup, which tips me toward accept.