ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching
We investigate the non-identifiability issues associated with bidirectional adversarial training for joint distribution matching. Within a framework of conditional entropy, we propose both adversarial and non-adversarial approaches to learn desirable matched joint distributions for unsupervised and supervised tasks. We unify a broad family of adversarial models as joint distribution matching problems. Our approach stabilizes learning of unsupervised bidirectional adversarial learning methods. Further, we introduce an extension for semi-supervised learning tasks. Theoretical results are validated in synthetic data and real-world applications.
Adversarially Learned Inference (a.k.a. Adversarial Feature Learning)
is an interesting extension to GANs, which can be used to train a
generative model by learning generator G(z) and inference E(x)
functions, where G(z) maps samples from a latent space to data and
E(x) is an inference model mapping observed data to the latent
space. This model is trained adversarially by jointly training E(x)
and G(z) with a discriminator D(x,z) which is trained to distinguish
between real (E(x), x) samples and fake (z, G(z)) samples. This is an
interesting approach and has been shown to generate latent
representations which are useful for semi-supervised learning.
The authors highlight an issue with the ALI model, by constructing a
small example for which there exist optimal solutions to the ALI loss
function which have poor reconstruction, i.e. G(E(x)) can be very
different to x. This identifiability issue is possible because the ALI
objective imposes no restrictions on the conditional distributions of
the generator and inference models. In particular, this allows for
solutions which violate a condition known as cycle consistency, which
is considered a desirable property in unsupervised learning. They
propose adding a cross-entropy term to the loss, which enforces a
constraint on one of the two conditionals.
The ALICE training objective is shown to yield better reconstructions
than ALI on toy data. They also evaluate on the inception score, a
metric proposed in Salimans et al. (2017), which also shows better 
performance for the model trained with the ALICE objective.
The description of how the model can be applied to semi-supervised
learning seemed a little strange to me. The authors consider a case
where the latent space z corresponds to the labels and semi-supervised
learning involves training a model with the ALICE objective, when some
paired (z,x) samples are available. They claim that these labelled
samples can be leveraged to address the identifiability issues in
ALI. While this setting holds for the experiments they present in
section 5.2, a more common application of this kind of model to
semi-supervised learning is to use the inference network to generate
features which are then used for classification. The results in the
ALI paper follow this approach and evaluate ALI for semi-supervised
learning on some standard benchmarks (CIFAR10 and SVHN). This is
possibly the strongest evidence in the ALI paper that the model is
learning a useful mapping from data space to latent space, so it would
be useful to see the same evaluation for ALICE.
In the ALI and BIGAN papers, it is shown that cycle consistency is
satisfied at the optimum when the E and G mappings are
deterministic. Similarly, in this paper the authors show that the
optimal solution to a model trained with cross-entropy regularisation
has deterministic mappings. In lines 228-233, the authors suggest that
a model trained with deterministic mappings is likely to underfit and
taking the alternative approach of training a model with stochastic
mappings and a cross-entropy regularisation gives faster and more
stable training. While this is an interesting claim, it was not
immediately clear to me why this would be the case from the
description provided in the paper.
This paper is on an interesting topic, highlighting a significant
issue with the ALI training objective and proposing a solution using
cross-entropy regularisation. It would be more convincing if the
authors had evaluated their model on the same set of experiments as
the original ALI paper, especially the semi-supervised learning
benchmarks, and provided more justification for the use of
cross-entropy regularisation over deterministic mappings