In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoencoder, and derive the learning rules based on maximum-likelihood learning. Using implicit distributions allows us to learn more expressive posterior and conditional likelihood distributions for the autoencoder. Learning an expressive conditional likelihood distribution enables the latent code to only capture the abstract and high-level information of the data, while the remaining information is captured by the implicit conditional likelihood distribution. For example, we show that implicit autoencoders can disentangle the global and local information, and perform deterministic or stochastic reconstructions of the images. We further show that implicit autoencoders can disentangle discrete underlying factors of variation from the continuous factors in an unsupervised fashion, and perform clustering and semi-supervised learning.
This paper introduces the implicit autoencoder, which purports to be a VAE with an implicit encoding and decoding distribution.
My principle problem with the paper and reason for my strong rejection is that there appears to be a complete separation between the discussion and theory of the paper and the actual experiments run.  The paper's discussion and theory all centers around rewriting the ordinary ELBO lower bound on the marginal likelihood in equations (4) through (7) where it is shown that this can be recast in the form of two KL divergences, one between the representational joint q(x,z) = p_data(x) encoder(z|x) and the 'reconstruction joint' r(x,z) = encoder_marginal(z) decoder(x|z), and one between the encoding marginal q(z) and the generative prior p(z).   The entire text of the paper then discusses the similarities between this formulation of the objective and some of the alternatives as well as discussing how this objective might behave in various limits.
However, this is not the objective that is actually trained. In the  "Training Process" section is it revealed that an ordinary GAN discriminator is trained.  The ordinary GAN objective does not minimize a KL divergence, it is a minimax formulation of a Jensen Shannon divergence as the original GAN paper notes.  More specifically, you can optimize a KL divergence with a GAN, as shown in the f-GAN paper (1606.00709) but this requires attention be paid to the functional form of the loss and structure of the discriminator.  No such care was taken in this case.  As such the training process does not minimize the objective derived or discussed.  Not to mention that in practice a further hack is employed wherein only the negative example passes gradients to the generator.  
While is not specified in the training process section, assuming the ordinary GAN objective (Equation 1) is used, according to their own reference (AVB) the optimal decoder should be:  D = 1/(1 + r(z,x)/q(z,x))  for which we have that what they deem the 'generative loss of the reconstruction GAN' is T = log(1 + r(z,x)/q(z,x))  .   When we take unbiased gradients of the expectation of this quantity, we do not obtain an unbiased gradient of the KL divergence between q(z,x) and r(z,x).
Throughout the paper, factorized Gaussian distributions are equated with tractable variational approximations.  While it is common to use a mean field gaussian distribution for the decoder in VAEs this is by no means required.  Many papers have investigated the use of more powerful autoregressive or flow based decoders, as this paper itself cites (van der Oord et al. 2016).  The text further misrepresents the current literature when it claims that the IAE uniquely "generalizes the idea of deterministic reconstruction to stochastic reconstruction by learning a decoder distribution that learns to match to the inverse encoder distribution".  All VAEs have employ stochastic reconstruction, if the authors again here meant to distinguish a powerful implicit decoder from a mean field gaussian one, the choice of language here is wrong.
Given that there are three joint distributions in equation (the generative model, the representational joint and the reconstruction joint), the use of Conditional entropy H(x|z) and mutual information I(x, z) are ambiguous.  While the particular joint distribution is implied by context in the equations, please spell it out for the reader.
The "Global vs. Local Decomposition of Information in IAEs" section conflates dimensionality with information capacity.  While these are likely correlated for real neural networks, at least fundamentally an arbitrary amount of information could be stored in even a 1 dimensional continuous random variable.  This is not addressed.  
The actual experiments look nice, its just that objective used to train the resulting networks is not the one presented in the paper. 
------
In light of the author's response I am changing my review from 2 to 3.  I still feel as though the paper should be rejected.  While I appreciate that there is a clear history of using GANs to target otherwise intractable objectives, I still feel like those papers are all very explicit about the fact that they are modifying the objective when they do so.  I find this paper confusing and at times erroneous.  The added appendix on the bits back argument for instance I believe is flawed.
"It first transmits z, which ideally would only require H(z) bits; however, since the code is designed
under p(z), the sender has to pay the penalty of KL(q(z)kp(z)) extra bits"
False.  The sender is not trying to send an unconditional latent code, they are trying to send the code for a given image, z \sim q(z|x).  Under usual communication schemes this would be sent via an entropic code designed for the shared prior at the cost of the cross entropy \int q(z|x) \log p(z) and the excess bits would be KL(q(z|x) | p(z)), not Kl(q(z)|p(z)).  
The appendix ends with "IAE only minimizes the extra number of bits required for transmitting x, while the VAE minimizes the total number of bits required for the transmission" but the IAE = VAE by Equation (4-6).  They are equivalent, how can one minimizing something the other doesn't?  In general the paper to me reads at times as VAE=IAE but IAE is better.  While it very well might be true that the objective trained in the paper (a joint GAN objective attempting to minimize the Jensen Shannon divergence between both  (1) the joint data density q(z,x) and the aggregated reconstruction density r(z,x) and (2) the aggregated posterior q(z) and the prior p(z)) is better than a VAE (as the experiments themselves suggest), the rhetoric of the paper suggests that the IAE referred to throughout is Equation (6).  Equation 6 is equivalent to a VAE.
I think the paper would greatly benefit from a rewriting of the central story.  The paper has a good idea in it, I just feel as though it is not well presented in its current form and worry that if accepted in this form might cause more confusion than clarity.  Combined with what I view as some technical flaws especially in the appendices I still must vote for a rejection.