While autoencoders are a key technique in representation learning for continuous structures, such as images or wave forms, developing general-purpose autoencoders for discrete structures, such as text sequence or discretized images, has proven to be more challenging. In particular, discrete inputs make it more difficult to learn a smooth encoder that preserves the complex local relationships in the input space. In this work, we propose an adversarially regularized autoencoder (ARAE) with the goal of learning more robust discrete-space representations. ARAE jointly trains both a rich discrete-space encoder, such as an RNN, and a simpler continuous space generator function, while using generative adversarial network (GAN) training to constrain the distributions to be similar. This method yields a smoother contracted code space that maps similar inputs to nearby codes, and also an implicit latent variable GAN model for generation. Experiments on text and discretized images demonstrate that the GAN model produces clean interpolations and captures the multimodality of the original space, and that the autoencoder produces improvements in semi-supervised learning as well as state-of-the-art results in unaligned text style transfer task using only a shared continuous-space representation.
I was asked to contribute this review rather late in the process, and in order
to remain unbiased I avoided reading other reviews. I apologize if some of
these comments have already been addressed in replies to other reviewers.
This paper proposes a regularization strategy for autoencoders that is very
similar to the adversarial autoencoder of Makhzani et al. The main difference
appears to be that rather than using the classic GAN loss to shape the
aggregate posterior of an autoencoder to match a chosen, fixed distribution,
they instead employ a Wasserstein GAN loss (and associated weight magnitude
constraint, presumably enforced with projected gradient descent) on a system
where the matched distribution is instead learned via a parameterized sampler
("generator" in the GAN lingo). Gradient steps that optimize the encoder,
decoder and generator are interleaved. The authors apply an extension of this
method to topic and sentiment transfer and show moderately good latent space
interpolations between generated sentences.
The difference from the original AAE is rather small and straightforward, making the
novelty mainly in the choice of task, focusing on discrete vectors and sequences.
The exposition leaves ample room for improvement. For one thing, there is the
irksome and repeated use of "discrete structure" when discrete *sequences* are
considered almost exclusively (with the exception of discretized MNIST digits).
The paper is also light on discussion of related work other than Makhzani et al
-- the wealth of literature on combining autoencoders (or autoencoder-like
structures such as ALI/BiGAN) and GANs merits at least passing mention.
The empirical work is somewhat compelling, though I am not an expert in this
task domain. The annealed importance sampling technique of Wu et al (2017) for
estimating bounds on a generator's log likelihood could be easily applied in
this setting and would give (for example, on binarized MNIST) a quantitative
measurement of the degree of overfitting, and this would have been preferable
than inventing new heuristic measures. The "Reverse PPL" metric requires more
justification, and it looks an awful lot like the long-since-discredited Parzen
window density estimation technique used in the original GAN paper.
High-level comments:
- It's not clear why the optimization is done in 3 separate steps. Aside
from the WGAN critic needing to be optimized for more steps, couldn't the
remaining components be trained jointly, with a weighted sum of terms for the
encoder?
- In section 2, "This [pre-training or co-training with maximum likelihood]
  precludes there being a latent encoding of the sentence." It is not at all
  clear to me why this would be the case.
- "One benefit of the ARAE framework is that it compresses the input to a
  single code vector." This is true of any autoencoder.
- It would be worth explaining, in a sentence, the approach in Shen et al for
  those who are not familiar with it, seeing as it is used as a baseline.
- We are told that the encoder's output is l2-normalized but the generator's
  is not, instead output units of the generator are squashed with the tanh
  activation. The motivation for this choice would be helpful. Shortly
  thereafter we are told that the generator quickly learns to produce norm 1
  outputs as evidence that it is matching the encoder's distribution, but this
  is something that could have just as easily have been built-in, and is a
  trivial sort of "distribution matching"
- In general, tables that report averages would do well to report error bars as
  well. In general some more nuanced statistical analysis of these results
  would be worthwhile, especially where they concern human ratings.
- The dataaset fractions chosen for the semi-supervised experience seem
  completely arbitrary. Is this protocol derived from some other source?
  Putting these in a table along with the results would improve readability. 
- Linear interpolation in latent space may not be the best choice here
  seeing as e.g. for a Gaussian code the region near the origin has rather low
  probability. Spherical interpolation as recommended by White (2016) may
  improve qualitative results.
- For the interpolation results you say "we output the argmax", what is meant?
  Is beam search performed in the case of sequences?
- Finally, a minor point: I will challenge the authors to justify their claim
  that the learned generative model is "useful" (their word). Interpolating
  between two sentences sampled from the prior is a neat parlour trick, but the
  model as-is has little utility. Even some speculation on how this aspect
  could be applied would be appreciated (admittedly, many GAN papers could use
  some reflection of this sort).