Deep generative models seek to recover the process with which the observed data was generated. They may be used to synthesize new samples or to subsequently extract representations. Successful approaches in the domain of images are driven by several core inductive biases. However, a bias to account for the compositional way in which humans structure a visual scene in terms of objects has frequently been overlooked. In this work we propose to structure the generator of a GAN to consider objects and their relations explicitly, and generate images by means of composition. This provides a way to efficiently learn a more accurate generative model of real-world images, and serves as an initial step towards learning corresponding object representations. We evaluate our approach on several multi-object image datasets, and find that the generator learns to identify and disentangle information corresponding to different objects at a representational level. A human study reveals that the resulting generative model is better at generating images that are more faithful to the reference distribution.
[Overview]
In this paper, the authors proposed a compositional image generation methods that combines multiple objects and background into the final images. Unlike the counterpart which compose the images sequentially, the proposed method infer the relationships between multiple objects through a relational network before sending the hidden vectors to the generators. This way, the method can model the object-object interactions during the image generation. From the experimental results, the authors demonstrated that the proposed k-GAN can generate the images with comparable or slightly better FID compared with baseline GAN, and achieve plausible performance under the human study.
[Strengthes]
1. This paper proposed an interesting method for compositional image generation. Unlike the counterparts like LR-GAN, which generate foreground objects recurrently, this method proposed to derive the relationships between objects in parallel simultaneously. This kind of relational modeling has been seen in many other domains. It would be nice to see it can be applied to the compositional image generation domain.
2. The authors tried multiple synthesized datasets, including multi-MNIST and its variants, CLEVR. From the visualization, it is found that the proposed k-GAN can learn to disentangle different objects and the objects from the background. This indicates that the proposed model indeed capture the hidden structure of the images through relational modeling. The human study on these generated images further indicate that the generated images based on k-GAN is better than those generated by baseline GAN.
[Weaknesses]
1. The main contribution of this paper fall to the proposed method for modeling the relational structure for multiple objects in the images. In the appendix, the authors presented the results for the ablated version which does not consider the relationships. As the authors pointed out, these results are a bit counterintuitive and concluded that FID is not a good metrics for evaluating compositional generation. However, as far as I know, the compositional generation can achieve much better Inception scores on CIFAR-10 in LR-GAN paper (Yang et al). Combining the results on MNIST in LR-GAN paper, I would suspect that the datasets used in this paper are fairly simple and all methods can achieve good results without much efforts. It would be good to show some results on more complicated datasets, such as face images with background, or cifar-10. Also, the authors did not present the qualitative results for independent version of k-GAN. Meanwhile, they missed an ablated human study when the relational modeling is muted. I would like to see how the generated images without modeling relationships look like to humans.
2. Following the above comment, I think the datasets used in this paper is relatively simpler. In MM and CLEVR, the foreground objects are digits or simple cubes, spheres or cylinders. Also, the background is also simpler for these two datasets. Though CIFAR10+MM has a more complicated background, it is trivial for the model to distinguish the foregrounds from the backgrounds. Again, the authors should try more complicated datasets.
3. Though the proposed method can model the relationship between objects simultaneously, I doubt its ability to  really being able to disentangle the foregrounds from the backgrounds. Since the background and foregrounds are both whole images, which are then combined with an alpha blending, the model cannot discover the conceptually different properties for foreground and background that foregrounds are usually small than background and scattered at various locations. Actually, this has been partially demonstrated by Figure 4. In the last row, we can find one sphere is in the background image. I tend to think the proposed model performs similar to Kwak & Zhang's paper without a strong proof for the authors that the relational modeling plays an important role in the model.
4. It would be nice to perform more analysis on the trained k-GAN. Given the training set, like MM or CLEVR, I am wondering whether k-GAN can learn some reasonable relationship from the datasets. That is, whether it is smart enough to infer the right location for each objet by considering the others. This analysis can be performed, how much occlusions the generated images have compared with the real images. For example, on CLEVR, I noticed from the appendix that the generated CLEVR images base on k-GAN actually have some severe occlusions/overlaps.
[Summary]
In this paper, the authors proposed an interesting method for image generation compositionally. Instead of modeling the generation process recurrently, the authors proposed to model the relationships simultaneously in the hidden vector space. This way, the model can generate multiple foreground objects and backgrounds more flexibly. However, as pointed above, the paper missed some experiment, ablation study and analysis to demonstrate the relational modeling in the image generation. The author need to either try more complicated images or add deeper analysis on the recent experimental results.