GENERATING MULTIPLE OBJECTS AT SPATIALLY DISTINCT LOCATIONS
Recent improvements to Generative Adversarial Networks (GANs) have made it possible to generate realistic images in high resolution based on natural language descriptions such as image captions. However, fine-grained control of the image layout, i.e. where in the image specific objects should be located, is still difficult to achieve. We introduce a new approach which allows us to control the location of arbitrarily many objects within an image by adding an object pathway to both the generator and the discriminator. Our approach does not need a detailed semantic layout but only bounding boxes and the respective labels of the desired objects are needed. The object pathway focuses solely on the individual objects and is iteratively applied at the locations specified by the bounding boxes. The global pathway focuses on the image background and the general image layout. We perform experiments on the Multi-MNIST, CLEVR, and the more complex MS-COCO data set. Our experiments show that through the use of the object pathway we can control object locations within images and can model complex scenes with multiple objects at various locations. We further show that the object pathway focuses on the individual objects and learns features relevant for these, while the global pathway focuses on global image characteristics and the image background.
The authors are proposing a method for allowing the generation of multiple objects in generated images given simple supervision such as bounding boxes and their associated labels. They control the spatial location of generated objects by the mean of an object pathway added to the architecture of both Generator and Discriminator within a GAN framework. They show generated results on Multi-MNIST, CLEVR with discussions of their model's abilities and properties. they also provide quantitative results on MSCOCO (IS and FID) using StackGAN and AttGAN models with the object pathway modifications and show some improvements compared to the original models. However it must be noted (as commented by the authors) that these models are using image captions only and do not have explicit supervision of bounding box and object labels.
This paper proposes a simple approach to generating requested objects in GAN-based image generation task, The method is supervised and requests (in its current form) the Bounding Boxes and Labels of the objects to integrate into the image generation. This task of controlling the nature (identity) and size of objects to integrate in a generated image is an important one and is significant to the GAN-base image generation community. In terms of originality, the approach is a nice simple architecture that takes care of the spatial location problem head-on. It seems like an obvious step but this does not take away from the merits of the proposed method.
The generator Global path is given a noise component. From the text, it does not seem that the Object path is given a noise component. Do you generate always the same object given the same label and Bounding Box then? Why not integrate some noise in this pipeline too?
Multi-MNIST:
The authors present results on Multi-MNIST 50K customed data to present the ability of the model to accurately put request images in the correct bounding box (BB) and do some ablation study. This is an interesting test as it shows that indeed the method proposed generates digits where it is expected to. Could you provide the ground truth labels for each/some image/s? For the failure cases it is often not clear what digit is what. For the Row E and F, 1s could be 7s and vice versa. Since it is a qualitative study, it would be nice to have the Ground Truth (GT) (which you provide to G at for generation). For the failure case of Row D (right) an interesting results would have been to have example of a digit bounding box from top to bottom with few pixel vertical shift to visualize when the model starts to mess-up the generation. This seems to point that your model (exposed to the location from BB for the object paths) is sensitive to what locations it has seen in training. How would you make the object path more robust to unseen location (overall you need to design an object of a given size, then locate it in your empty canvas prior to the CNN for generation)?
CLEVR:
The images resolution make it hard to really see the shape of the images (here too, the GT would be great). The bounding boxes make the images even harder to parse. I know the colors change but "We can confirm that the model can control both location and object's shape". For the location, it is true, for the shape is hard to completely tell at this resolution without GT. 
MS-COCO:
 Just a comment in passing on the fact that resizing images from COCO to 256x256 will inherently distort quite a bit of images, the median size (for each dim) for COCO is 640x480, if I am not mistaken. Most, if not all images in COCO are not 1.0 size ratio.
The quantitative results on COCO seem to confirm that the proposed method is generating "better" images according to IS and FID. This is a good thing, however the technique is strongly supervised (Bounding Box and Object Labels, caption compared to solely captions for StackGAN and AttGAN) so this result should be expected and really put into perspective as your are not comparing models w/ the same supervision (which you mention in the Discussion).
Discussion: 
I appreciate that the authors addressed the limitations of their approach in this section. The overlapping BBs seems to be an interesting challenge. Did you try to normalize the embeddings in overlapping area? A simple sum does not seem to be a good solution. In Figure 7 w/ overlapping zebras, the generation seems completely lost. 
In terms of clarity, the paper is well-written but would benefit *greatly* from using variables names when discussing 'layout embedding', 'generated local labels', etc. Variable names and equations, while not necessary, can go a long way to clearly express a model's internal blocks (most of the papers you referenced are using this approach). The paper employs none of this commonly used standard and suffers from it. I myself had to write down on the margin the different variables used at each step described in text to have an understanding of what was done (with help of Figure 1). You should reference Figure 1 in the Introduction, as you cover your approach there and the Figure is useful to grasp your contributions.
Another comment concerning clarity is, while it is fine to rely on previously published papers for description of our own work, you should not assume full knowledge from the reader and your paper should stand on its own without having the reader lookup for several papers to have an understanding of your training procedures. If one uses GAN training, it should be expected to cover/formulate quickly the min max game and the various losses you are trying to minimize. I am afraid that "using common GAN procedures" is not enough. When describing your experimental setup, pointing to another paper as "hyperparameters remain the same as in the original training procedure" should not be a substitution for covering it too, even if lightly in the Appendix. For instance: in the Appendix, it is mentioned that training was stopped at 20 epochs for Multi-MNIST, 40 for CLEVR... How did you decide on the epoch (early stopping, stopped before instabillity of GAN training, etc.) Did you use SGD? ADAM? Did you adjust the learning rate, which schedule? etc. for your GAN training. This information in the Appendix would make the paper overall stronger. 
Last comment: In terms of generation multiple objects. Have you had the chance to run an object detector on your generated image (you can build one on MSCOCO given the bounding box and label, finetune an ImageNet pretrained model). It would be interesting to see if the generated images are good enough for object detection.
Post-Rebuttal: Given the work from the authors on improving the clarity of the paper as well as investigating the use of object detection metrics to compare their methods, I decided to move my rating upward to 7