Generative adversarial networks (GANs) are a powerful framework for generative tasks. However, they are difficult to train and tend to miss modes of the true data generation process. Although GANs can learn a rich representation of the covered modes of the data in their latent space, the framework misses an inverse mapping from data to this latent space. We propose Invariant Encoding Generative Adversarial Networks (IVE-GANs), a novel GAN framework that introduces such a mapping for individual samples from the data by utilizing features in the data which are invariant to certain transformations. Since the model maps individual samples to the latent space, it naturally encourages the generator to cover all modes. We demonstrate the effectiveness of our approach in terms of generative performance and learning rich representations on several datasets including common benchmark image generation tasks.
This paper presents the IVE-GAN, a model that introduces en encoder to the Generative Adversarial Network (GAN) framework. The model is evaluated qualitatively through samples and reconstructions on a synthetic dataset, MNIST and CelebA.
Summary: 
The evaluation is superficial, no quantitative evaluation is presented and key aspects of the model are not explored. Overall, there just is not enough innovation or substance to warrant publication at this point.
Impact:
The motivation given throughout the introduction -- to add an encoder (inference) network to GANs -- is a bit odd in the light of the existing literature. In addition to the BiGAN/ALI models that were cited, there are a number of (not cited) papers with various ways of combining GANs with VAE encoders to accomplish exactly this. If your goal was to improve reconstructions in ALI, one could simply add an reconstruction (or cycle) penalty to the ALI objective as advocated in the (not cited) ALICE paper (Li et al., 2017 -- "ALICE: Towards Understanding Adversarial Learning for Joint Distribution Matching"). 
The training architecture presented here is novel as far as I know, though I am unconvinced that it represents an optimum in model design space. The model presented in the ALICE paper would seem to be a more elegant solution to the motivation given in this paper.
Model Feature:
The authors should discuss in detail the interaction between the regular GAN pipeline and the introduced variant (with the transformations). Why is the standard GAN objective thrown in? I assume it is to allow you to sample directly from the noise in z (as opposed to z' which is used for reconstruction), but this is not discussed in much detail. The GAN objective and the added IVE objective seem like they will interact in not all together beneficial ways, with the IVE component pushing to make the distribution in z complicated. This would result in a decrease in sample quality. Does it? Exploration of this aspect of the model should be included in the empirical evaluation.
Also the addition of the transformations added to the proposed IVE pipeline seem to cause the latent variations z' to encode these variations rather than the natural variations that exist in the dataset. They would seem to make it difficult to encode someone face and make some natural manipulates (such as adjusting the smile) that are not included in this transformations.
Empirical Evaluation:
Comparison to BiGAN/ALI: The authors motivate their work by drawing comparisons to BiGAN/ALI, showing CelebA reconstructions from the ALI paper in the appendix. The comparison is not fair for two reasons, (1) authors should state that their reconstructions are made at a higher resolution (seems like 128x128, which is now standard but was not so when the BiGAN/ALI papers came out, they were sampled at 64x64), also, unlike the ALI results, they authors cut the background away from the CelebA faces. This alone could account for the difference between the two models, as ICE-GAN only has to encode the variability extant in faces and hair, ALI had to additionally encode the much greater variability in the background. The failure to control the experimental conditions makes this comparison inappropriate. 
There is no quantitative evaluations at all. While many GAN papers do not place an emphasis on quantitative evaluations, at this point, I consider the complete lack of such an evaluation as a weakness of the paper. 
Finally, based on just the samples offered in the paper, which is admittedly a fairly weak standard, the model does not seem to be among the state-of-the-art on CelebA that have been reported in the literature. Given the rapid progress that is being made, I do not feel this should be could against this particular paper, but the quality of samples cannot be considered a compelling reason to accept the paper.
Minor comment:
The authors appear to be abusing the ICLR style file by not leaving a blank line  between paragraphs. This is annoying and not at all necessary since ICLR does not have a strict page limit. 
Figure 1 is not consistent with the model equations (in Eqns. 3). In particular, Figure 1 is missing the standard GAN component of the model.
I assume that the last term in Eqns 3 should have G(z) as opposed to G(z',E(x)). Is that right?