Non-Adversarial Mapping with VAEs
The study of cross-domain mapping without supervision has recently attracted much attention. Much of the recent progress was enabled by the use of adversarial training as well as cycle constraints. The practical difficulty of adversarial training motivates research into non-adversarial methods. In a recent paper, it was shown that cross-domain mapping is possible without the use of cycles or GANs. Although promising, this approach suffers from several drawbacks including costly inference and an optimization variable for every training example preventing the method from using large training sets. We present an alternative approach which is able to achieve non-adversarial mapping using a novel form of Variational Auto-Encoder. Our method is much faster at inference time, is able to leverage large datasets and has a simple interpretation.
The manuscript considers the problem of unsupervised mapping across domains by use of variational auto-encoders (VAE) instead of the current adversarial training (GAN) approach.  The approach extends a recently proposed non-adversarial mapping (NAM) framework extracting a mapping from a pretrained generative model to a target domain with the present addition of a probabilistic encoder which maps the target image y to the latent domain z_y, i.e. based on the optimization given in equation 2 of the manuscript.  Quality: The issue of mapping between domains is interesting and the framework presented seems new and useful. It however also appears to be a straightforward extension to NAM but providing computational merits and improved performance in terms of the quality of mapping which could warrant publication. The experimentation is somewhat limited and the presentation of the paper could be improved by proofreading the manuscript.  Clarity: The basic idea of VAE-NAM appears intuitive, i.e. from a generative model of X formed by G(z) create an encoder that encodes Y in this latent space z_y=E(y) such that a transformation T of the corresponding X resembles Y, i.e. minimize ||T(G(E(y))),y|| as defined in equation 2 treating the encoder with uncertainty using the framework of VAE.  However, the manuscript could be clearer in its presentation. First of all the encoder is called C throughout the manuscript and in equation 2 called E which causes confusion. Furthermore, second section of the VAE-NAM description is unclear. Should x be y in the first two lines, i.e. line 167 and 168? In section 4.1 line 234 is referenced to table 1 I believe this should be Figure 1. There is further several spelling issues, i.e., is key component -> is a key component analogies the can -> analogies that can  this was found at preserving -> this was found to preserve aforemntioned -> aforementioned direct to code -> directly to code very recently began to generate -> very recently begun to generate  Originality: The manuscript appears to be a straightforward extension of NAM to include a variational autoencoder and the work appears somewhat incremental.  Significance The results are compelling and the procedure seem to have merits both computationally and in quality of results that may warrant publication. Avoiding the adversarial training and the challenges imposed there also makes the current framework attractive with a potential impact.   I thank the authors for their response and find the paper more compelling given their response to the issues raised. I have accordingly increased my evaluation of the paper from just below to marginally above the acceptance threshold.