Generative image models have made significant progress in the last few years, and are now able to generate low-resolution images which sometimes look realistic. However the state-of-theart models utilize fully entangled latent representations where small changes to a single neuron can effect every output pixel in relatively arbitrary ways, and different neurons have possibly arbitrary relationships with each other. This limits the ability of such models to generalize to new combinations or orientations of objects as well as their ability to connect with more structured representations such as natural language, without explicit strong supervision.
In this work explore the synergistic effect of using partial natural language scene descriptions to help disentangle the latent entities visible an image. We present a novel neural network architecture called Generative Entity Networks, which jointly generates both the natural language descriptions and the images from a set of latent entities. Our model is based on the variational autoencoder framework and makes use of visual attention to identify and characterise the visual attributes of each entity. Using the Shapeworld dataset, we show that our representation both enables a better generative model of images, leading to higher quality image samples, as well as creating more semantically useful representations that improve performance over purely dicriminative models on a simple natural language yes/no question answering task.
**Summary**
The paper proposes an extension of the attend, infer, repeat generative model of Eslami, 2016 and extends it to handle ``visual attribute descriptions. This straightforward extension is claimed to improve image quality and shown to improve performance on a previously introduced image caption ranking task. In general, the paper shows improvements on an image caption agreement task introduced in Kuhnle and Copestake, 2017.  The paper seems to have weaknesses pertaining to the approach taken, clarity of presentation and comparison to baselines which mean that the paper does not seem to meet the acceptance threshold for ICLR. See more detailed points below in Weaknesses.
**Strengths**
I like the high-level motivation of the work, that one needs to understand and establish that language or semantics can help learn better representations for images. I buy the premise and think the work addresses an important issue. 
**Weakness**
Approach:
* A major limitation of the model seems to be that one needs access to both images and attribute vectors at inference time to compute representations which is a highly restrictive assumption (since inference networks are discriminative). The paper should explain how/if one can compute representations given just the image, for instance, say by not using amortized inference. The paper does propose to use an image-only encoder but that is intended in general as a modeling choice to explain statistics which are not captured by the attributes (in this case location and orientation as explained in the Introduction of the paper).
Clarity:
* Eqn. 5, LHS can be written more clearly as \hat{a}_k. 
* It would also be good to cite the following related work, which closely ties into the model of Eslami 2016, and is prior work: 
Efficient inference in occlusion-aware generative models of images,
Jonathan Huang, Kevin Murphy.
ICLR Workshops, 2016
* It would be good to clarify that the paper is focusing on the image caption agreement task from Kuhnle and Copestake, as opposed to generic visual question answering.
* The claim that the paper works with natural language should be toned down and clarified. This is not natural language, firstly because the language in the dataset is synthetically generated and not “natural”. Secondly, the approach parses this “synthetic” language into structured tuples which makes it even less natural. Also, Page. 3. What does “partial descriptions” mean?
* Section 3: It would be good to explicitly draw out the graphical model for the proposed approach and clarify how it differs from prior work (Eslami, 2016).
* Sec. 3. 4 mentions that the “only image” encoder is used to obtain the representation for the image, but the “only image” encoder is expected to capture the “indescribable component” from the image, then how is the attribute information from the image captured in this framework? One cannot hope to do image caption association prediction without capturing the image attributes...
*, In general, the writing and presentation of the model seem highly fragmented, and it is not clear what the specifics of the overall model are. For instance, in the decoder, the paper mentions for the first time that there are variables “z”, but does not mention in the encoder how the variables “z” were obtained in the first place (Sec. 3.1). For instance, it is also not clear if the paper is modeling variable length sequences in a similar manner to Eslami, 2016 or not, and if this work also has a latent variable [z, z_pres] at every timestep which is used in a similar manner to Eqn. 2 in Eslami, 2016. Sec. 3.4 “GEN Image Encoder” has some typo, it is not clear what the conditioning is within q(z) term.
* Comparison to baselines: 
  1. How well does this model do against a baseline discriminative image caption ranking approach, similar to [D]? This seems like an important baseline to report for the image caption ranking task.
  2. Another crucial baseline is to train the Attend, Infer, Repeat model on the ShapeWorld images, and then take the latent state inferred at every step by that model, and use those features instead of the features described in Sec. 3.4 “Gen Image Encoder” and repeat the rest of the proposed pipeline. Does the proposed approach still show gains over Attend Infer Repeat?
  3. The results shown in Fig. 7 are surprising -- in general, it does not seem like a regular VAE would do so poorly. Are the number of parameters in the proposed approach and the baseline VAE similar? Are the choices of decoder etc. similar? Did the model used for drawing Fig. 7 converge? Would be good to provide its training curve. Also, it would be good to evaluate the AIR model from Eslami, 2016 on the same simple shapes dataset and show unconditional samples. If the claim from the work is true, that model should be just as bad as a regular VAE and would clearly establish that using language is helping get better image samples.
* Page 2: In general the notion of separating the latent space into content and style, where we have labels for the “content” is an old idea that has appeared in the literature and should be cited accordingly. See [B] for an earlier treatment, and an extension by [A]. See also the Bivcca-private model of [C] which has “private” latent variables for vision similar to this work (this is relevant to Sec. 3.2.)
References:
[A]: Upchurch, Paul, Noah Snavely, and Kavita Bala. 2016. “From A to Z: Supervised Transfer of Style and Content Using Deep Neural Network Generators.” arXiv [cs.CV]. arXiv. http://arxiv.org/abs/1603.02003.
[B]: Kingma, Diederik P., Danilo J. Rezende, Shakir Mohamed, and Max Welling. 2014. “Semi-Supervised Learning with Deep Generative Models.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1406.5298.
[C]: Wang, Weiran, Xinchen Yan, Honglak Lee, and Karen Livescu. 2016. “Deep Variational Canonical Correlation Analysis.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1610.03454.
[D]: Kiros, Ryan, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. “Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/1411.2539.