Adversarial Scene Editing: Automatic Object Removal from Weak Supervision
While great progress has been made recently in automatic image manipulation, it has been limited to object centric images like faces or structured scene datasets. In this work, we take a step towards general scene-level image editing by developing an automatic interaction-free object removal model. Our model learns to find and remove objects from general scene images using image-level labels and unpaired data in a generative adversarial network (GAN) framework. We achieve this with two key contributions: a two-stage editor architecture consisting of a mask generator and image in-painter that co-operate to remove objects, and a novel GAN based prior for the mask generator that allows us to flexibly incorporate knowledge about object shapes. We experimentally show on two datasets that our method effectively removes a wide variety of objects using weak supervision only.
The details provided on the experiment with SDI[2] are somewhat lacking, and the question on how that works combined with standard inpainting was not really addressed. However, they raise a valid point in regards to it requiring box-level supervision compared to the image-level supervision they use. Overall I'm still leaning to accept this submission.  ====================================== The paper proposes a system that can learn to remove objects of specific categories from images automatically (e.g. persons). The system is composed of two stages: 1)  a mask generator which zero-es out regions in the image such that an image classifier does not recognize the specified category (e.g. it is not classified as person) 2) an inpainting network which infills the masked image so that it looks realistic again  Notably, the mask network uses weak supervision: it only can see real masks from a smaller dataset through a GAN loss. More specifically, it is trained to produce masks that hide the target object (fooling an image level classifier) and to also produce masks that are realistic from the view of a mask discriminator.   Overall the paper is well written and provides extensive experiments, building a strong baseline for the task from Mask-RCNN + inpainting network (which uses stronger supervision since mask-rcnn is trained on a large corpus of segmentation masks). Qualitatively the network works fairly well, and quantitatively it is competitive with the mask-rcnn baseline. The benefit of the weak supervision is demonstrated by showing that it can work on a dataset that has no segmentation masks available (a logo dataset).  The paper makes the point that it is actually helpful to have coarse masks that hide the (to be removed) object's boundaries, compared to fine grained masks. This is demonstrated both when comparing the proposed approach (which has coarse masks because of weak supervision) to the mask-rcnn baseline, and also by improving the mask-rcnn baseline by dilating its segmentation masks.  However, based on this I question the need for the the GAN based mask generator. One could interpret the results as that for object removal, you don't really need a very good instance segmentation network (since dilating mask-rcnn helps). Therefore, we should also expect to see good results when using a weakly supervised instance segmentation network. In principle, we can view the first stage of the system as exactly that, but shouldn't the baseline for weak supervision then be as follows:  * Standard weakly supervised instance segmentation network, such as (Simple Does It: Weakly Supervised Instance and Semantic Segmentation, Khoreva et al, CVPR 2017 ). * Possibly dilate its masks * Use standard inpainting approach on top  So basically, I question the need for the first stage: is it not just a weakly supervised instance segmentation approach? If so, is competitive with established such approaches, and should they not be the baseline?  Another concern is what will the whole system be used for? I understand that removing people from images in an unsupervised manner is a difficult problem, but it would be nice to see a concrete example where it is useful.  Nontheless, I found the paper overall interesting in terms of the problem studied, the approach and the quality of the experiments, so I'm currently leaning towards accepting it.