Image Inpainting via Generative Multi-column Convolutional Neural Networks
In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.
Summary This submission tackles the problem of image inpainting. It adapts the idea of multi-scale predictions by running three branches that predict features at different scales which are concatenated before two convolutions create the final prediction. The loss consists of three components. The relatively common L1 and local+global discriminator loss and lastly a novel implicitly diversified MRF that correlates patches of features between the ground truth and the prediction. The method is evaluated on five diverse datasets and with several ablation studies analyzing the influence of different parts.  Strengths  - The ablation study shows the contribution of the different components well. I am quite surprised about the added detail through the increased receptive field.   - The method shows impressive image quality on a wide range of datasets.  - The paper is technically sound, well structured and easy to follow.  - The fact that the expensive MRF optimization can be pushed to the training and is not needed for evaluation is elegant and makes the model much more usable during inference.   Weaknesses  - Since it is difficult to evaluate image generation tasks quantitatively, a human study could be performed where users need to select their preferred generated image. This evaluation could help showing the increased visual quality with respect to other methods.  - Since Equation 5 is applied multiple times it should contain some notion of iterations (e.q. M_w^{(i)}. Otherwise M_w appears on both sides of the equation.   - An interesting evaluation could be correlating reconstruction quality and RS measure. This could show that the loss actually helps finding the right correspondences between patches.  Minor Comments  - L. 82: {1, 2, …, n}  - The type of upsampling for the feature maps before concatenation is not mentioned.   - Since the ID-MRF is only used in training it would be interesting to report training times or even time per image during training compared to inference.   - Potential typo in Equation (4). L_M(conv4_2) appears twice.   - The intuitive example for L_M from the supplementary material could be included in the paper. I found it easier to follow than the explanations in lines 117-128.  - L. 224: our method is still difficult to deal with large-scale -> still has difficulties dealing with […]   Rating & Evaluation Given the above strengths and weaknesses I am in favor of accepting this submission. A user study and a more in-depth analysis of the proposed MRF loss could make this paper even stronger.   --  After reading the other reviews and the rebuttal I find most of my concerns addressed. The authors provide several additional quantitative results including a user study with convincing outcome. I vote in favor of accepting the paper.