Invariant Representations without Adversarial Training
Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation. Unfortunately, learning representations that exhibit invariance to arbitrary nuisance factors yet remain useful for other tasks is challenging. Existing approaches cast the trade-off between task performance and invariance in an adversarial way, using an iterative minimax optimization. We show that adversarial training is unnecessary and sometimes counter-productive; we instead cast invariant representation learning as a single information-theoretic objective that can be directly optimized. We demonstrate that this approach matches or exceeds performance of state-of-the-art adversarial approaches for learning fair representations and for generative modeling with controllable transformations.
The paper studies the timely and extremely relevant problem of learning invariant representations for data. In contrary to the main stream of works in this area, it does not consider an adversarial training approach, and this is refreshing! Instead, it proposes to solve an information-theoretic objective, which proves to be competitive and even better that state-of-the-art adversarial approaches.   In more details, the authors derive a useful upper-bound on the mutual information, that can be nicely integrated in the VAE or VIB frameworks. This derivation is essentially the main contribution of the paper, and it provides an elegant and generic framework for learning invariant representations.   The paper is generally well written, even it is quite dense and probably too detailed at places, like in Section 2. This sort of leads the reader to lose the main intuition behind the proposed framework.   The experiments are generally well chosen, and the visualisation of Fig 2 is pretty interesting and original. The results are however not always fully convincing (see for example the German dataset in Fig 1), but it does not critically reduce the quality of the paper and the proposed ideas.   Suggestions:  - the title 'evading the adversary' is a bit misleading: what is done in the paper is actually learning invariant representations... - while there is an attempt to stay generic in the presentation of the framework, it sometimes leads to confusion: in particular, the mix between fairness ideas, and more classical invariant representation learning objectives is sometimes confusing (like in the related work section, and in the experiments).   Overall, the paper addresses a timely problem, with a new approach, based on information theoretic concepts and bounds on the mutual information. These ideas are integrated into classical learning architectures, and leads to competitive results.