Pose Guided Person Image Generation
This paper proposes the novel Pose Guided Person Generation Network (PG 2 ) that allows to synthesize person images in arbitrary poses, based on an image of that person and a novel pose. Our generation framework PG 2 utilizes the pose information explicitly and consists of two key stages: pose integration and image refinement. In the first stage the condition image and the target pose are fed into a U-Net-like network to generate an initial but coarse image of the person with the target pose. The second stage then refines the initial and blurry result by training a U-Net-like generator in an adversarial way. Extensive experimental results on both 128×64 re-identification images and 256×256 fashion photos show that our model generates high-quality person images with convincing details.
The paper proposes a human image generator conditioned on appearance and human pose. The proposed generation is based on adversarial training architecture where two-step generative networks that produces high resolution image to feed into a discriminator. In the generator part, the first generator produce a coarse image using a U-shape network given appearance and pose map, then the second generator takes the coarse input with the original appearance to predict residual to refine the coarse image. The paper utilizes the DeepFashion dataset for evaluation.
The paper proposes a few important ideas.
* Task novelty: introducing the idea of conditioning on appearance and pose map for human image generation
* Techniques: stacked architecture that predicts difference map rather than direct upsampling, and loss design
The paper can improve in terms of the following points to stand out.
* Still needs quality improvement
* Significance: the paper could be seen one of yet-another GAN architecture
* Problem domain: good vision/graphics application, but difficult to generalize to other learning problems
The paper is well organized to convey the key aspects of the proposed architecture. Conditioned on appearance and pose information, the proposed generator stacks two networks to adopt a coarse-to-fine strategy. This paper effectively utilize the generation strategy in the dressing problem. The proposed approach looks appropriate to the concerned problem scenario. The difference map generation also looks a small but nice technique in generating higher resolution images.
Probably the major complaints to the paper is that the generated results contain visible artifacts and still requires a lot of improvement for application perspective. For example, patterns in ID346 of Fig 4 results in black dots in the final result. Even though the second generator mitigates the blurry image from the first generator, it seems the model is still insufficient to recover high-frequency components in the target appearance.
Another possible but not severe concern is that some might say the proposed approach is an application of conditional GANs. Conditioning or stacking of generators for adversarial training have been proposed in the past; e.g., below, though they are arXiv papers. The paper includes application-specific challenges, but this might not appeal to large number of audiences.
* Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, Dimitris Metaxas, "StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks", arXiv:1612.03242.
* Xun Huang, Yixuan Li, Omid Poursaeed, John Hopcroft, Serge Belongie, Stacked Generative Adversarial Networks, arXiv:1612.04357.
In overall, the paper successfully proposed a solution to the pose-conditioned image problem, and properly conducts evaluation. The proposed approach sufficiently presents technical novelty. The resulting images still needs quality improvement, but the proposed model at least generate something visually consistent images. My initial rating is accept.