Learning Hierarchical Semantic Image Manipulation through Structured Representations
Understanding, reasoning, and manipulating semantic concepts of images have been a fundamental research problem for decades. Previous work mainly focused on direct manipulation on natural image manifold through color strokes, keypoints, textures, and holes-to-fill. In this work, we present a novel hierarchical framework for semantic image manipulation. Key to our hierarchical framework is that we employ structured semantic layout as our intermediate representation for manipulation. Initialized with coarse-level bounding boxes, our structure generator first creates pixel-wise semantic layout capturing the object shape, object-object interactions, and object-scene relations. Then our image generator fills in the pixel-level textures guided by the semantic layout. Such framework allows a user to manipulate images at object-level by adding, removing, and moving one bounding box at a time. Experimental evaluations demonstrate the advantages of the hierarchical manipulation framework over existing image generation and context hole-filing models, both qualitatively and quantitatively. Benefits of the hierarchical framework are further demonstrated in applications such as semantic object manipulation, interactive image editing, and data-driven image manipulation.
In this paper a new method for image manipulation is proposed. The proposed method incorporates a hierarchical framework and provides both interactive and automatic semantic object-level image manipulation. In the interactive manipulation setting, the user can select a bounding box where image editing for adding and removing objects will be applied. The proposed network architecture consists of a foreground output stream which produces the predictions on binary object mask and a background output stream for producing per-pixel label maps. As the result, the proposed image manipulation method generates output image by filling in the pixel-level textures guided by the semantic layout. The manipulation operations are only ‘insertion’ and ‘deletion’ and other operations such as change of color and style are not supported.   The paper is well-written and clear; Figures and qualitative examples convey the idea. As to investigate the effectiveness of the single-stream vs two-stream using ground truth or predicted layout ablation study is conducted.   The performance of the proposed method is compared with prior works of [21] and [32] and human study is conducted via AMT. Based on the results summarized in Table 2, the proposed method has higher SSIM and obtains higher human evaluation compared to the prior works and baseline. However, as for the the segmentation accuracy the prior work of [32] obtains higher accuracy.   While the proposed method is interesting, the evaluation is only conducted on outdoor street images and only two selected examples are shown in the indoor scenes in supplemental. In light of the high quality performance of the previous works on image generation such as [4] and [41], image generation in the context of indoor scenes is more subject to evaluation as such indoor scenes involve more visual diversity and contain larger set of object categories for manipulation.  Most of the examples shown in the paper demonstrate the quality of adding or removing cars and person in street scenes which have high biases in terms of pose and shape. The question that remains is that how much object diversity (in terms of both category and appearance differences) such image manipulation technique can handle.