Unsupervised Adversarial Invariance
Data representations that contain all the information about target variables but are invariant to nuisance factors benefit supervised learning algorithms by preventing them from learning associations between these factors and the targets, thus reducing overfitting. We present a novel unsupervised invariance induction framework for neural networks that learns a split representation of data through competitive training between the prediction task and a reconstruction task coupled with disentanglement, without needing any labeled information about nuisance factors or domain knowledge. We describe an adversarial instantiation of this framework and provide analysis of its working. Our unsupervised model outperforms state-of-the-art methods, which are supervised, at inducing invariance to inherent nuisance factors, effectively using synthetic data augmentation to learn invariance, and domain adaptation. Our method can be applied to any prediction task, eg., binary/multi-class classification or regression, without loss of generality.
The paper presents a general procedure to disentangle the factors of variation in the data that contribute to the prediction of a desired target variable and nuisance factors of variation. The authors achieve this by partitioning the hidden representations learned by neural network “h” into “e1” that contains factors that contribute directly to predicting the outcomes “y” and “e2” that contains nuisance factors that are not useful in predicting “y”. An adversarial disentanglement objective is used to ensure that e1 and e2 capture different pieces of information and a data reconstruction objective (with e1 noised) is used to ensure that all of the information is not present solely in e1 while e2 learns nothing.  The overall approach is clever, sound and well thought out. The motivation of disentanglement and learning nuisance factors in data is well presented as well.  My main concerns with this paper is the experimental work:  1) The paper claims to outperform methods where the nuisance factors are known in advance and can be used learn representations that are “explicitly” invariant to these factors via data augmentation or adversarial training. However, I think the baselines presented that use this information are not strong enough, for example in MNIST where different rotations or other affine transformations are applied to the data, how does this compare to the spatial transformer network? (Jaderberg et. al 2016).  2) I found almost all presented tasks to be toyish in nature, while it is certainly useful to experiment in settings where the nuisance factors are known in advance, I would encourage the authors to also work with richer datasets where an adversarial disentanglement objective, a classification objective and a reconstruction objective, working together can be hard to train. Specifically, I see the reconstruction objective becoming harder to optimize with high dimensional inputs. Demonstrating the capability of this model to work with richer datasets will help with adoption. Simple MNIST and YALE-B experiments are a good start but are insufficient to demonstrate clear improvements.  3) It would be informative to also report in Table 1 the accuracy of predicting y from e2 (by “post-fitting” a classifier after training) and z from e2. Along the same lines, more ablations that add different amounts of noise to e1 during reconstruction, alter the coefficients on the disentanglement versus reconstruction objectives would be useful to report.  In conclusion I think that this is a promising piece of work that currently lacks strong experimental backing. I'd be more convinced by experiments on harder and richer datasets.