Soft-Gated Warping-GAN for Pose-Guided Person Image Synthesis
Despite remarkable advances in image synthesis research, existing works often fail in manipulating images under the context of large geometric transformations. Synthesizing person images conditioned on arbitrary poses is one of the most representative examples where the generation quality largely relies on the capability of identifying and modeling arbitrary transformations on different body parts. Current generative models are often built on local convolutions and overlook the key challenges (e.g. heavy occlusions, different views or dramatic appearance changes) when distinct geometric changes happen for each part, caused by arbitrary pose manipulations. This paper aims to resolve these challenges induced by geometric variability and spatial displacements via a new Soft-Gated Warping Generative Adversarial Network (Warping-GAN), which is composed of two stages: 1) it first synthesizes a target part segmentation map given a target pose, which depicts the region-level spatial layouts for guiding image synthesis with higher-level structure constraints; 2) the Warping-GAN equipped with a soft-gated warping-block learns feature-level mapping to render textures from the original image into the generated segmentation map. Warping-GAN is capable of controlling different transformation degrees given distinct target poses. Moreover, the proposed warping-block is lightweight and flexible enough to be injected into any networks. Human perceptual studies and quantitative evaluations demonstrate the superiority of our Warping-GAN that significantly outperforms all existing methods on two large datasets.
The authors propose a computer vision framework for performing pose transfer, i.e. generating images of a given person in a random pose. The key technical contributions of this approach are 1) exploiting an external human parser for an intermediate step of predicting target segmentations given the input and the target pose, 2) a warping module estimating transformations between the input and the predicted target human parsings.   In my opinion, the paper is rather interesting and original: I like the idea of explicitly learned transformations and its adaptation for the task. There have been a number of works around modeling deformations (for example, [23] estimated local affined transformations for body parts), but the proposed implementation went further by learning transformation fields densely and end-to-end. The experimental results show significant improvements over the state-of-the-art given the chosen standard benchmarks and the metrics.  The manuscript is rather easy to follow and understand, but it still contains a significant number of typos and minor grammar mistakes. Some implementation details are missing which may make this work difficult to reproduce. It would be helpful if the authors provided a more detailed description of the geometric matcher.   My high level concern, however, is that there has been a large number of similar works optimizing for performance (overfitting?) on DeepFashion/Market-1501 datasets, which have rather low variety and cannot be considered representative of real data. I personally would be excited to see us as a community move beyond these benchmarks and target more ambitious settings.   My more detailed comments and questions are below.  The human parser has been trained on LIP dataset constructed from COCO images - therefore, I'd like to see some comments on the quality of obtained segmentations (on DeepFashion, and especially on the low resolution Market-1501 dataset) and how it affects the quality of the generations.  Stage 1 & Figure 2. Does the input to the trained parser include the original RGB image? L130-131 suggest so, while Figure 2 doesn't. Are heatmaps and probability maps concatenated? How is the background label treated?  Figure 3. Is the Geometric Matcher module trained separately or jointly with the rest of the generator? What is the meaning of the block on the right of the Matching layer? Adding labels to this figure would be helpful.  Geometric Matcher & Warping Block. The abbreviation TPS is used before it's defined.  What makes the network generalize on the body parts that are not observed in the source image (for example, going from front to back)? The description of the warping block would be easier to understand if the authors showed the corresponding notations in Figure 4. Why is learning facilitated by decomposing the result into the original set of features and residuals? How is the network's performance affected if the warping is done directly? The term "soft gate" is unclear in this context and its position in Figure 4 is not obvious.  Discriminator architecture. In the proposed implementation, the discriminator takes an exhaustive set of inputs (input image, target keypoints, source parsing, target parsing, generated image) - have the authors analyzed the importance of each of these components?  Objective functions. It would be easier for the reader if the authors explicitly described their loss terms (L1 or L2 for L_pixel, base network for L_perceptual, the exact kind of GAN objective). Stabilizing discriminators by matching intermediate features at multiple scales has been done before - the authors may want to cite: Wang et al. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. CVPR, 2018.  Datasets. How is the parser used to remove failure cases from the dataset, assuming that there is no ground truth for this data?  How is the training / test splits are defined? The number of training / test images in this paper differs from, for example, [23]. Assuming it was arbitrary, did the authors retrain the state-of-the-art models on the same split? Otherwise, the comparison may be unfair (images from the training set used for evaluation).  Symbol beta is overloaded (used in Eq. (2) and also to define Adam's parameters). How are betas in (2) are defined?  Using Inception score has been a common place in similar works, but one should keep in mind that it's not strictly appropriate for the task and in practice is highly unstable (can be high for "blurry" images obtained with L1/L2 loss). I'm not sure what the authors mean saying that high IS scores confirm generalization (L239). Also, it would be useful to report IS for the real images.  Human study: what was the exact protocol of the experiments? (number of images, number of workers, number of experiments / worker, how the images were selected, image resolution). It is somewhat disappointing that in these experiments the workers were asked to evaluate the "realism" of the generations, but not how well the target pose is captured or whether the identity and the appearance of the person from the original image are preserved. It would be interesting to see more results on these aspects as well. Why doesn't Table 2 contain comparison of the proposed method's outputs with ground truth images?   Nitpicking:  L119: we ARE following L136: please reformulate "parsing obey the input pose" L142: number of pixelS L162: Obtained -> Having obtained? L197: please rephrase "large arrange among of poses" L228: Figure shown -> Figure shows L246: significantLY bettter L268: generator significantly improveS Paper formatting seems to have suffered by reducing the manuscript to 8 pages, please fix.