We propose a novel autoencoding model called Pairwise Augmented GANs. We train a generator and an encoder jointly and in an adversarial manner. The generator network learns to sample realistic objects. In turn, the encoder network at the same time is trained to map the true data distribution to the prior in latent space. To ensure good reconstructions, we introduce an augmented adversarial reconstruction loss. Here we train a discriminator to distinguish two types of pairs: an object with its augmentation and the one with its reconstruction. We show that such adversarial loss compares objects based on the content rather than on the exact match. We experimentally demonstrate that our model generates samples and reconstructions of quality competitive with state-of-the-art on datasets MNIST, CIFAR10, CelebA and achieves good quantitative results on CIFAR10.
Thank you for an interesting read.
The paper proposes adding an adversarial loss to improve the reconstruction quality of an auto-encoder. To do so, the authors define an auxiliary variable y, and then derive a GAN loss to discriminate between (x, y) and (x, decoder(encoder(x))). The algorithm is completed by combining this adversarial "reconstruction" loss with adversarial loss functions that encourages the matching of marginal distributions for both the observed variable x and the latent variable z. 
Experiments present quite a lot of comparisons to existing methods as well as an ablation study on the proposed "reconstruction" loss. Improvements has been shown on reconstructing input images with significant numbers.
Overall I think the idea is new and useful, but is quite straight-forward and has some theoretical issues (see below). The propositions presented in the paper are quite standard results derived from the original GAN paper, so for that part the contribution is incremental and less interesting. The paper is overall well written, although the description of the augmented distribution r(y|x) is very rush and unclear to me.
There is one theoretical issue for the defined "reconstruction" loss (for JS and f-divergences). Because decoder(encoder(x)) is a deterministic function of x, this means p(y|x) is a delta function. With r(y|x) another delta function (even that is not delta(y=x)), with probability 1 we will have mismatched supports between p(y|x) and r(y|x). 
This is also the problem of the original GAN, although in practice the original GAN with very careful tuning seem to be OK... Also it can be addressed by say instance noise or convolving the two distributions with a Gaussian, see [1][2].
I think another big issue for the paper is the lack of discussion on how to choose r(y|x), or equivalently, a(x). 
1. Indeed matching p_{\theta}(x) to p^*(x) and q(z) to p(z) does not necessarily returns a good auto-encoder that makes x \approx decoder(encoder(x)). Therefore the augmented distribution r(y|x) also guides the learning of p(y|x) and with appropriately chosen r(y|x) the auto-encoder can be further improved.
2. The authors mentioned that picking r(y|x) = \delta(y = x) will result in unstable training. But there's no discussion on how to choose r(y|x), apart from a short sentence in experimental section "...we used a combination of reflecting 10% pad and the random crop to the same image size...". Why this specific choice? Since I would imagine the distribution r(y|x) has significant impact on the results of PAGAN, I would actually prefer to see an in-depth study of the choice of this distribution, either theoretically or empirically. 
In summary, the proposed idea is new but straight-forward. The experimental section contains lots of results, but the ablation study by just removing the augmentation cannot fully justify the optimality of the chosen a(x). I would encourage the authors to consider the questions I raised and conduct extra study on them. I believe it will be a significant contribution to the community (e.g. in the sense of connecting GAN literature and denoising methods literature).
[1] Sonderby et al. Amortised MAP Inference for Image Super-resolution. ICLR 2017.
[2] Roth et al. Stabilizing Training of Generative Adversarial Networks through Regularization. NIPS 2017.