Controllable Invariance through Adversarial Feature Learning
Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning. In this paper, we tackle the problem of learning representations invariant to a specific factor or trait of data. The representation learning process is formulated as an adversarial minimax game. We analyze the optimal equilibrium of such a game and find that it amounts to maximizing the uncertainty of inferring the detrimental factor given the representation while maximizing the certainty of making task-specific predictions. On three benchmark tasks, namely fair and bias-free classification, language-independent generation, and lighting-independent image classification, we show that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance.
The paper proposes to learn invariant features using adversarial training. Given s a nuisance factor (s attribute of x), a discriminator tries to predict the nuisance factors s  (an attribute of the input) given a encoder representation h=E(x,s)  , and an encode rE  tries to minimize  the prediction of nuisance factor and and to predict the desired output. The encoder is function of x and s.
Novelty:
The paper draws some  similarity with Ganian et al on unsupervised domain adaptation and their JMLR version. The applications to  Multilingual  machine translation and fairness applications  are to the best of the knowledge of the reviewer new in this context and are  interesting. 
Comments:
- The application of the method to invariant features learning in object recognition is not realistic. In machine translation s being the one hot of the language at hand or in fairness s being an attribute of the input , is realistic. But if one wants invariance to rotation or lightning one can not supply the angle or the lightening to the encoder. E(x,s) is not realistic for geometric invariance i.e when s is indeed hidden and not given at test time. Have you tried in this case for instance E being just E(x)?
See for instance this work  https://arxiv.org/pdf/1612.01928.pdf  that uses actually a very similar training with just E(x)
- I think the paper should be more focused on the multiple sources / single target domain adaptation aspect. Geometric invariances are confusing in this setting, since the nuisance factor can not be assumed known at test time.
- How would you extend this to deal with continuous nuisances factors s?