UNDERSTANDING AND IMPROVING INTERPOLATION IN AUTOENCODERS VIA AN ADVERSARIAL REGULARIZER
Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code. In some cases, autoencoders can "interpolate": By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints. In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefficient from interpolated data. We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting. We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations.
Summary: The authors propose a new approach to encourage valid interpolation in Auto-Encoders (AE). It is based on a regularization procedure involving a critic network judging the realistic nature of reconstructed data point from its mixed latent representations by recovering the mixing coefficient. The authors show that this approach does indeed improve the quality of interpolated samples on few tasks. A synthetic tasks of lines interpolation (proposing new Mean Distance and Smoothness metric for this task), classification task (with a single-layer classifier) from the latent space representation and finally a clustering accuracy on the latent space. On the proposed regularization method seems to help significantly compared to commonly used AE architectures (Basic AE, Denoising AE, Variational AE, Adversarial AE and VQ-VAE).
This paper was a very interesting read, and the work seems to be of significance for the unsupervised learning community.
It was clearly written and conveys the contributions clearly and the experimental results and their interpretations seem valid.
The proposed approach of a critic based regularizer is a simple but seemingly important addition that contributes to improving interpolation in AE significantly and even show impact "downstream tasks" as the authors put it.
Few comments/questions come to mind:
- For the critic Loss L_d in equation (1) , the authors mention that the \gamma based second term (that should ensure that the critic outputs 0 for non-interpolated inputs and expose the critic to realistic data even if the AE reconstruction is poor)  does not seem to be crucial in your approach but stabilized the adversarial training. Could you somehow quantify this. It seems like stability of the adversarial training should be paramount to your method to make sure the AE learns a better latent representation. This comment, even though I assume it well-founded, seems a bit of a contradiction.
- For the Lines synthetic data. It was chosen to use a 32x32 image size with 16 points length lines. This configuration does quantize directly the angles your measures can distinguish. Below a certain angle differences (or delta), 2 angles must have the same pixel representation, i.e. exact overlapping lines. My question is simple: What is the smallest angle you can use/distinguish or, how many exact unique lines can you have? 
Overall this is a good paper that deserves publications.