Context-Aware Synthesis and Placement of Object Instances
Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem. Solving it requires (a) determining a location to place an object in the scene and (b) determining its appearance at the location. Such an object insertion model can potentially facilitate numerous image editing and scene parsing applications. In this paper, we propose an end-to-end trainable neural network for the task of inserting an object instance mask of a specified class into the semantic label map of an image. Our network consists of two generative modules where one determines where the inserted object mask should be (i.e., location and scale) and the other determines what the object mask shape (and pose) should look like. The two modules are connected together via a spatial transformation network and jointly trained. We devise a learning procedure that leverage both supervised and unsupervised data and show our model can insert an object at diverse locations with various appearances. We conduct extensive experimental validations with comparisons to strong baselines to verify the effectiveness of the proposed network. Code is available at https: //github.com/NVlabs/Instance_Insertion.
In overall, I think this paper proposes a well-designed pipeline for semantic map synthesis which is helpful to image generation tasks. The rebuttal addresses my questions about the ablation study. I strongly suggest the authors to add the ablation study in the final version if this paper is accepted. My final rating is accept. ========================================= This paper proposes a where and what pipeline to synthesize and place object instances into a semantic map in an end-to-end way. This method can help to generate diverse natural semantic map which contains important structure information. The generated semantic maps can benefit the image synthesis task.  Pros: * The proposed idea is interesting that using the context information to predict the position and shape of object instances in order to generate novel semantic maps. * The writing is easy to follow, and empirical experiments show that it is able to synthesize natural semantic maps. * The decomposed where and what modules make scenes and can provide some control during the synthesis. The well-designed jointly training strategy is suitable for the semantic map synthesis task. The proposed framework can also be useful for other structure generation tasks.  Cons: * In total, there are four discriminators. I wonder how do they affect the training and results. Is there any ablation study for this?  Given that this paper is about the context-aware location and shape prediction, it is suggested that some recent work on context based structure prediction for image inpainting and generation can be included in the related work, e.g.,  [a] Q. Sun, L. Ma, S.J. Oh, L. Van Gool, B. Schiele, and M. Fritz. “Natural and Effective Obfuscation by Head Inpainting.” In CVPR, 2018. [b] K. Ehsani, R. Mottaghi, and A. Farhadi. “Segan: Segmenting and generating the invisible.” In CVPR 2018.