The use of adequate feature representations is essential for achieving high performance in high-level human cognitive tasks in computational modeling. Recent developments in deep convolutional and recurrent neural networks architectures enable learning powerful feature representations from both images and natural language text. Besides, other types of networks such as Relational Networks (RN) can learn relations between objects and Generative Adversarial Networks (GAN) have shown to generate realistic images. In this paper, we combine these four techniques to acquire a shared feature representation of the relation between objects in an input image and an object manipulation action description in the form of human language encodings to generate an image that shows the resulting end-effect the action would have on a computer-generated scene. The system is trained and evaluated on a simulated dataset and experimentally used on real-world photos.
1. The paper aims to train a model to move objects in an image using language. For instance, an image with a red cube and blue ball needs to be turned into an image  with a red cube and red ball if asked to "replace the red cube with a blue ball". The task itself is interesting as it aims to modify system behavior through language. 
The approach the authors  take is to encode an image with a CNN, encode the sentence with an RNN and use both representations to reconstruct  the frame (via a relational network and decoder) that solves this task. This process as described was already done in (Santoro 2017). The idea of using a CNN feature map and LSTM embedding to solve spatial reasoning tasks is not new. 
The main contribution is to add a discriminative loss to turn the problem into a "is this solution correct or not." This is interesting but does not perform much better than the baseline of not using the GAN loss (as suggested by the results in Table 2). This suggests that the GAN term is not adding as much value as the authors claim.
2. Reject
- Reason 1: The results in Table 2 show that the GAN does slightly better (0.0134 vs 0.0144) in RMSE against the non-GAN version. This improvement does not seem statistically significant enough to warrant the added GAN complexity.
- Reason 2: Other baselines need to be considered, AE, VAE or other variations.
- Reason 3: No ablations on the impact of the parameters to eq 1.
3. To improve the paper I suggest adding other baselines such as VAE, AE. In addition, consider using more negative samples instead of the single negative image.