## Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation

## Hao Tang

DISI, University of Trento hao.tang@unitn.it Trento, Italy

Dan Xu danxu@robots.ox.ac.uk

Gaowen Liu g\_l198@txstate.edu Texas State University San Marcos, USA

## Wei Wang

wei.wang@epfl.ch EPFL Lausanne, Switzerland

University of Oxford Oxford, UK

Nicu Sebe niculae.sebe@unitn.it DISI, University of Trento Trento, Italy

## ABSTRACT

Input

In this work, we propose a novel Cycle In Cycle Generative Adversarial Network (C 2 GAN) for the task of keypoint-guided image generation. The proposed C 2 GAN is a cross-modal framework exploring a joint exploitation of the keypoint and the image data in an interactive manner. C 2 GAN contains two different types of generators, i.e., keypoint-oriented generator and image-oriented generator. Both of them are mutually connected in an end-to-end learnable fashion and explicitly form three cycled sub-networks, i.e., one image generation cycle and two keypoint generation cycles. Each cycle not only aims at reconstructing the input domain, and also produces useful output involving in the generation of another cycle. By so doing, the cycles constrain each other implicitly, which provides complementary information from the two different modalities and brings extra supervision across cycles, thus facilitating more robust optimization of the whole network. Extensive experimental results on two publicly available datasets, i.e., Radboud Faces [19] and Market-1501 [58], demonstrate that our approach is effective to generate more photo-realistic images compared with state-of-the-art models.

## CCS CONCEPTS

· Computing methodologies → Artificial intelligence ; Computer vision ; Machine learning .

## KEYWORDS

Generative Models; Generative Adversarial Networks (GANs); Object Keypoint; Facial Landmark; Image-to-Image Translation; Human Pose Generation; Facial Expression Generation.

## ACMReference Format:

Hao Tang, Dan Xu, Gaowen Liu, Wei Wang, Nicu Sebe, Yan Yan. 2019. Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.

MM'19, October 21-25, 2019, Nice, France

ACM ISBN 978-1-4503-6889-6/19/10...$15.00

© 2019 Association for Computing Machinery.

https://doi.org/10.1145/3343031.3350980

Yan Yan y\_y34@txstate.edu Texas State University San Marcos, USA

GT Keypoint

C2GAN

GT Image

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-29fe807e-063d-49e2-b1c4-4d9c4f6dd08b.jpg)

Figure 1: The proposed C GAN on two challenging tasks, 2 i.e., landmark-guided facial expression generation (Top) and keypoint-guided person pose generation (Bottom). From left to right: input image, ground truth target keypoint, ground truth target image and C GAN (ours). 2

Generation. In Proceedings of the 27th ACM International Conference on Multimedia (MM '19), October 21-25, 2019, Nice, France. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3343031.3350980

## 1 INTRODUCTION

Humans have the ability to convert objects or scenes to another form just by imagining, while it is difficult for machines to deal with this task. For instance, we can easily generate mental images that have different facial expressions and human poses. In this paper, we study how to enable machines to perform image-to-image translation tasks, which has many application scenarios, such as human-computer interactions, entertainment, virtual reality and data augmentation. One important benefit of this task is that it can help to augment training data by generating diverse images with given input images, which thus could be employed to improve other recognition or detection tasks.

However, the task is still challenging since: (i) it needs to handle complex backgrounds with different illumination conditions, objects and occlusions; (ii) it needs a high-level semantic understanding of the mapping between the input images and the output images since the objects in the inputs may have arbitrary poses, sizes, locations and self-occlusions. Recently, Generative Adversarial Networks (GANs) [11] have shown the potential to solve this difficult task, and it can be utilized, for instance, to convert a face with a neutral expression into different expressions or to transfer a person with a specific pose into different poses. GANs have produced promising results in many generative tasks, such as photo-realistic image generation [3, 13, 15, 60], video generation [5, 24, 38, 48, 49, 54], text generation [56], audio generation [47] and image inpainting [9, 57]. Recent works have developed powerful image translation systems, e.g., Pix2pix [13] and Pix2pixHD [50] in supervised settings, where image pairs are required. However, paired training data are usually difficult and expensive to obtain. To tackle this problem, CycleGAN [60], DualGAN [55] and DiscoGAN [16] provide an interesting insight, in which the models can learn the mapping from one image domain to another with unpaired data. However, these models encounter the efficiency issue. For instance, with m different image domains, CycleGAN, DiscoGAN, DualGAN need to train m m ( - ) 1 generators and discriminators. While Pix2pix has to train m m ( - ) 1 generator/discriminator pairs. Recently, Anoosheh et al. propose ComboGAN [2], which only needs to train m generator/discriminator pairs in term of m different image domains. Tang et al. [46] propose G 2 GAN, in which a dual-generator and a discriminator can perform unpaired image-to-image translation for multiple domains. In addition, Choi et al. [7] propose StarGAN, a single generator/discriminator pair can perform unpaired image-to-image translation for multiple domains. While the computational complexity of StarGAN is Θ ( 1 , this model is not ) effective in handling some specific image-to-image translation tasks such as person image generation [20, 39] and hand gesture generation [43], in which image generation could involve infinity image domains m since human body and hand gesture in the wild can have arbitrary poses, sizes, appearances and locations.

Current state-of-the-art keypoint-guided image translation methods such as PG 2 [20] and PoseGAN [39] have two main issues: (i) both only directly transfer from an original domain to a target domain, without considering the mutual translation between each other, while the translation across different modalities in a joint network would bring rich cross-modal information. (ii) both simply employ the keypoint information as input reference to guide the generation, without involving the generated keypoint information as supervisory signals to further improve the network optimization. Both issues lead to unsatisfactory results.

To address these limitations, several works are proposed to generate images based on object keypoints or human skeletons. Keypoint/skeleton contains the object information of shapes and position, which can be used to produce more photo-realistic images. For instance, Reed et al. [34] propose GAWWN model, which generates bird images conditioned on both text descriptions and object location. Qiao et al. [31] present GCGAN to generate facial expression conditioned on geometry information of facial landmarks. Song et al. [40] propose G2GAN for facial expression synthesis. Siarohin et al. [39] introduce PoseGAN for pose-based human image generation. Tang et al. [43] propose GestureGAN for skeleton-guided hand gesture generation. Ma et al. [20] propose PG 2 , which can generate person images using a conditional image and a target pose. An illustrative comparison among PG 2 [20], PoseGAN [39] and the proposed C 2 GAN is shown in Fig. 2. PG 2 tries to generate person images using target keypoints L y . For PoseGAN, which needs the target keypoints L y and original keypoints L x as conditional inputs. Both methods only employ keypoint information as input guidance.

To address these limitations, we propose a novel Cycle In Cycle Generative Adversarial Network (C 2 GAN), in which explicitly three cycled sub-networks are formed to learn the image translation crossing modalities in a unified network structure. We have a basic image cycle, i.e., I2I2I ( [ x , L y ] GI → [ y ∗ , L x ] GI → x ∗ ), which aims at reconstructing the input and further refine the generated images y ∗ . The keypoint information in C 2 GAN is not only utilized as input guidance but also act as output, meaning that the keypoint is also a generative objective. Input and output of the keypoint are connected by two keypoint cycles, i.e., K2G2K ( [ x , L y ] → GI y ∗ GK → L ∗ y ) and K2R2K ( [ y ∗ , L x ] → GI x ∗ GK → L ∗ x ), where G I and G K denotes an image and a keypoint generator, respectively. In this way, keypoint cycles can provide weak supervision to the generated images y ∗ . The intuition of the keypoint

Figure 2: An illustrative comparison of existing models (a) PG 2 [20], (b) PoseGAN [39] and (c) the proposed C GAN. 2 BothPG andPoseGANonlyusekeypointasinputguidance, 2 while C GAN is a cross-modal model in which keypoint in2 formation not only acts as input guidance but also acts as output. The different cycles provide cross-modality complementary information and crossing cycle supervision for better network optimization.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-8168a1a6-e52d-43e7-9f4c-34f03db952c5.jpg)

cycles is that if the generated keypoint is very close to the real keypoint, then the corresponding images should be similar. In other words, better keypoint generation will boost the image generation, and conversely the improved image generation can facilitate the keypoint generation. These three cycles inherently constraint each other in the network optimization in an end-to-end training fashion.

In summary, the contribution of this paper is three-fold:

Moreover, for better optimization the three cycles we propose two novel cycle losses, i.e., Image Cycle-consistency loss (IC) and Keypoint Cycleconsistency loss (KC). With these cycle losses, each cycle can benefit from each other in joint learning. Moreover, we propose two cross-modal discriminators corresponding to the generators. We conduct extensive experiments on two different keypoint-guided image generation tasks, i.e., landmarkguided facial expression generation and keypoint-guided person pose generation. Extensive experimental results demonstrate that C 2 GAN yields superior performance compared with state-of-the-art approaches.

- · We propose a novel cross-modal generative adversarial network named Cycle In Cycle Generative Adversarial Network (C 2 GAN) for keypoint-guided image generation task, which organizes the keypoint and the image data in an interactive generation manner in a joint deep network, instead of using the keypoint information only as a guidance for the input.
- · Extensive results on two challenging tasks, i.e., landmark-guided facial expression generation and keypoint-guided person pose generation demonstrate the effectiveness of the proposed C 2 GAN, and show more photo-realistic image generation compared with existing competing models.
- · The cycle in cycle structure is a new network design which explores effective utilization of cross-modal information for the keypointguided image generation task. The designed cycled sub-networks connect different modalities, and implicitly constraint on each other, leading to extra supervision signals for better image generation. We also investigate cross-modal discriminators and cycle losses for more robust network optimization.

## 2 RELATED WORK

Generative Adversarial Networks (GANs) [11] have shown the capability of generating high-quality images [3, 12, 14, 15, 52]. Although it is successful in many tasks, it also has many challenges, such as how to control the content of the generated images. To generate meaningful images that meet user requirement, Conditional GAN (CGAN) [26] is proposed where the conditioned information is employed to guide the image generation process. A CGAN model always combines a vanilla GAN and an external information, such as discrete class labels or tags [10, 28, 30], text descriptions [23, 32, 34] semantic maps [29, 35, 44, 50], conditional images [13], object masks [27] or attention maps [6, 22, 25, 45]. However, existing CGANs synthesize images based on global constraints such as a class label, text description or facial attribute, they do not provide control over pose, object location or object shape.

However, existing paired and unpaired image translation approaches are inefficient and ineffective as discussed in the introduction section. Most importantly, these aforementioned approaches cannot handle some specific image-to-image translation tasks such as person image generation problem [20, 39], which could have infinity image domains since a person can have arbitrary poses, sizes, appearances and locations in the wild.

Image-to-Image Translation models use input-output data to learn a parametric mapping between inputs and outputs, e.g., Isola et al. [13] propose Pix2pix, which employs a CGAN to learn a mapping function from input to output image domains. Wang et al. [50] introduce Pix2pixHD model for synthesizing high-resolution images from semantic label maps. However, most of the tasks in the real world suffer from the constraint of having few or none of the paired input-output samples available. To overcome this limitation, the unpaired image-to-image translation task has been proposed. Different from the prior works, unpaired image-to-image translation task learns the mapping function without the requirement of paired training data, such as [2, 16, 42, 45, 46, 55, 59, 60]. For instance, Zhu et al. [60] introduce CycleGAN framework, which achieves unpaired image-to-image translation using the cycle-consistency loss. DualGAN is demonstrated in [55], in which there are image translators to be trained from two unlabeled image sets each representing an image domain. Kim et al. [16] propose a method based on GANs that learns to discover relations between different domains.

Keypoint-guided Image-to-Image Translation. To address these aforementioned limitations, several works [8, 20, 33, 39, 54] have been proposed to generate images based on object keypoint. For instance, Di et al. [8] propose GPGAN to synthesize faces based on facial landmarks. Reed et al. [33] present PixelCNN model to generate images part keypoints and text descriptions. Korshunova et al. [18] use facial keypoints to define the affine transformations of the alignment and realignment steps for face swap. Wang et al. [51] propose CMM-Net for landmark-guided smile generation. Sun et al. [41] propose a two-stage framework to perform head inpainting conditioned on the generated facial landmark in the first stage. Chan et al. [5] propose a method to transfer motion between human subjects based on pose stick figures in different videos. Yan et al. [54] propose a method to generate human motion sequence with a simple background using CGAN and human skeleton information.

The aforementioned approaches focus on a single image generation task. However, in this paper, we propose a novel Cycle In Cycle Generative Adversarial Network (C 2 GAN) which is a multi-task model and aims to handle two different tasks using one single network, i.e., image and keypoint generation. During the training stage, two tasks are restricted mutually by three cycles and benefits from each other. To the best of our knowledge, the proposed model is the first attempt to generate both the image and the keypoint domain in an interactive generation manner within a unified cycle in cycle GAN framework, for the keypoint-guided image translation task. Training GANs are a complicated optimization task and incorporating adversarial keypoint in training provides extra deep supervision to the image generation network compared to using supervision only from the image domain, thus facilitating the network optimization. Moreover, the keypoint generation aims not only to approximate the ground truth output but also to fool the discriminator, meaning that the generated keypoints should represent a real face or a person pose. The correlations between these keypoints can be learned in the adversarial setting.

## 3 CYCLE IN CYCLE GAN (C GAN) 2

We start to present the proposed Cycle in Cycle Generative Adversarial Network (C 2 GAN). Firstly, we introduce the network structures of the three different image and keypoint cycles, and also describe details for the corresponding generators and cross-modal discriminators. Secondly, the proposed objective functions for better optimization of the model and also will be illustrated, and finally the implementation details of the whole model and the training procedure are introduced.

## 3.1 Model Overview

The goal of the proposed C 2 GAN is to learn two different generators in one single network, i.e., keypoint generator and image generator. Two generators are mutually connected through three generative adversarial cycles, i.e., one image-oriented cycle and two keypoint-oriented cycles. In the training stage, all the cycled sub-networks are jointly optimized in an end-to-end fashion and each generator benefit from each other due to the richer cross-modal information and the crossing cycle supervision. The core framework of the proposed C 2 GAN is illustrated in Fig. 3. In the following, we describe the structure details of the proposed C 2 GAN.

## 3.2 Image-Domain Generative Adversarial Cycle

I2I2I Cycle. The goal of the image cycle I2I2I is to (i) generate image y ∗ by using the input conditional image x and the target keypoint L y , and then (ii) reconstruct the input image x by using the generated image y ∗ and the keypoint L x of image x . I2I2I cycle can be formulated as:

$$[ x, L _ { y } ] \stackrel { G _ { I } } { \to$$

where G I is the image generator. Different from previous works such as PG 2 [20] and PoseGAN [39], which only have one mapping [ x , L y ] GI → y ∗ . StarGAN [7] uses the target and original domain labels l y and l x as condition information to recover the input image. However, StarGAN can only handle the task which has a specific number of the category. For the person pose generation task, which could have infinity image domains since a person in the wild can have arbitrary poses, sizes, appearances and locations. In order to solve this limitation, we replace domain labels l y and l x in StarGAN by the keypoint L y and L x . Follow PG 2 we represent the keypoint as heatmaps. We concatenate x and L y and feed them into the image generator G I to generate y ∗ . Next, we concatenate y ∗ and L x as inputs of G I to reconstruct the original image x . In this way, the forward and backward consistency can be enforcedly further guaranteed.

Cross-modal Image Discriminator. Different from previous works such as PG 2 [20] which employs a single-modal discriminator, we propose a novel cross-modal discriminator which receives both keypoint and image data as input. D I receives two images and one keypoint data as input. More specifically, D I aims to distinguish between the generated triplet [ x , L y , G I ( x , L y )] and the real triplet [ x , L y , y ] during image generation

Image Generator. We use the U-net architecture [36] for our image generator G I . U-net is comprised of encoder and decoder with skip connections between them. We use G I two times for generating image y ∗ and reconstructing image x ∗ . To reduce model capacity, generators G I shares parameters between image generation and reconstruction. For image generation, the target of G I is to generate an image y ∗ = G I ( x , L y ) conditioned on the target keypoint image L y which is similar to the real images y . For image reconstruction, the goal of generator G I is to recover an image x ∗ = G I ( y ∗ , L x ) that looks close to the input images x . G I tries to learn a combined data distribution between the generation and the reconstruction by sharing parameters, which means G I receives double data in optimization compared to the generators without using parameter sharing strategy.

Figure 3: Framework overview of the proposed C GAN. It contains two types of generators, i.e., image generator 2 G I and keypoint generator G K . Parameter-sharing strategies can be used in between the image or the keypoint generators to reduce the model capacity. In the training stage, two generators are explicitly connected by three cycles, i.e., the image cycle I2I2I: [ x , L y ] → [ G I y ∗ , L x ] → G I x ∗ and two keypoint cycles K2G2K: [ x , L y ] → G I y ∗ G K → ∗ L y , K2R2K: [ y ∗ , L x ] → G I x ∗ G K → ∗ L x . The right side of the figure shows different cross-modal discriminators for better network optimization.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-b77cf417-96b7-448b-a976-f2dac7824246.jpg)

stage. We also propose an image adversarial loss L I GAN ( G I , D I , x , y , L y ) based on the vanilla adversarial loss [11]. The image adversarial loss can be formulated as follows:

## 3.3 Keypoint-Domain Generative Adversarial Cycle

$$\mathcal { L } ^ { I } _ { G A N } ( G _ { I }$$

G I tries to minimize L I GAN ( G I , D I , x , y , L y ) while D I tries to maximize it. A similar image adversarial loss for image reconstruction mapping G I : [ y ∗ , L x ] → x ∗ is defined as:

$$\begin{array} {$$

$$\Phi _ { \text{data} }$$

where D I aims at distinguishing between the fake triplet [ y , L x , G I ( y ∗ , L x )] and the real triplet [ y , L x , x ] . Thus the overall image adversarial loss is the sum of Eq. (2) and Eq. (3):

$$& \mathcal { L } ^ { I } _ { G A N } ( G _ { I$$

Image Cycle-Consistency Loss. To better learn the image cycle I2I2I, we propose an image cycle-consistency loss. The loss function writes:

$$& \mathcal { L } _ { C Y C } ^ { I } ( G _ { I }, x, L _ { x }, L _ { y } ) = & \quad \begin{pmatrix} & & \quad \begin{pmatrix} & \quad \end{pmatrix} \\ & \overbrace { \text{sho} } \\ & \overbrace { \text{cycl} } \end{pmatrix}$$

The reconstructed images x ∗ = G I ( G I ( x , L y ) , L x ) should closely match to the input image x . Note that we use generator G I two times with the parameter-sharing strategy, and we use L 1 distance in image cycleconsistency loss to compute a pixel-to-pixel difference between the recovered image x ∗ and the real input image x .

The motivation of the keypoint cycle is that, if the generated keypoint is similar to the real keypoint then the corresponding two images should be very close, as we can see in Fig. 3. We have two keypoint cycles K2G2K and K2R2K. Both of them can provide a supervision signal for optimizing better the image cycle.

K2G2K Cycle. For the K2G2K cycle, we feed [ x , L y ] into the image generator G I to produce the target image y ∗ . Then we employ the keypoint generator G K to produce the keypoint image L ∗ y from y ∗ . The generated keypoint L ∗ y should be very close to the real keypoint image L y . The formulation of K2G2K can be expressed as:

$$[ x, L _ { y } ] \stackrel { G _ { I } } { \to } y ^ { * } \stackrel { G _ { K } } { \to } L _ { y } ^ { * }.$$

K2R2K Cycle. For K2R2K cycle, the generated image y ∗ and keypoint image L x are first concatenated, and then feed into G I to produce the recovered image x ∗ . We use G K to generate the keypoint image L ∗ x of x ∗ . We assume that the generated keypoint L ∗ x is very similar to the real keypoint image L x . For the K2R2K cycles, it can be formulated as:

$$[ y ^ { * }, \, L _ { x } ] \stackrel { G _ { I } } { \to } x ^ { * } \stackrel { G _ { K } } { \to } L _ { x } ^ { * }.$$

Both generated keypoints L ∗ y = G K ( G I ( x , L y )) and L ∗ x = G K ( G I ( y ∗ , L x )) should have a close match to the input keypoint image L y and L x , respectively. Note that the generator G K could share parameters between the two cycles, i.e., K2G2K and K2R2K.

Keypoint Generator. We employ U-net structure [36] for our keypoint generator G K . The input of G K is an image and the output is a keypoint representation. The keypoint generator produces keypoint L ∗ y = G K ( y ∗ ) and L ∗ x = G K ( x ∗ ) from image y ∗ and x ∗ , which can provide extra supervision to the image generator.

Cross-Modal Keypoint Discriminator. The proposed keypoint discriminator D K is a cross-modal discriminator. It receives both image and keypoint data as inputs. Thus the keypoint adversarial loss for D K can be defined as:

$$\mathcal { L } _ { G A N } ^ { K } ( G _ { K }, D _ { K }, y ^ { * }, L _ { y } ) = \begin{array} { c } \underset { \text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text	$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text
$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text  $\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{	$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\font$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text#\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{#\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text$$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\test$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text}$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text'$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text`$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text.$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text1$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text<$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$\text{$$$

G K tries to minimize the keypoint adversarial loss L K GAN ( G K , D K , y ∗ , L y ) while D K tries to maximize it. D K aims to distinguish between the fake pair [ y ∗ , L ∗ y ] and the real pair [ y ∗ , L y ] . A similar keypoint adversarial loss for the mapping function G K : x ∗ → ∗ L x is defined as:

$$\cdots \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \ tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau\tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tut \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \varepsilon } \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \delta } \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau
\cdots \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \varepsi } \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau\ tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \t. \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \sigma } \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau. \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau \tau.$$

where discriminator D K aims to distinguish between the fake pair [ x ∗ , L ∗ x ] and the real pair [ x ∗ , L x ] . Thus, the total keypoint adversarial loss is the sum of Eq. (8) and Eq. (9):

$$& \mathcal { L } _ { G A N } ^ { K } ( G _ { K }, D _ { K }, x ^ { * }, y ^ { * }, L _ { x }, L _ { y } ) = \\ & \mathcal { L } _ { G A N } ^ { K } ( G _ { K }, D _ { K }, y ^ { * }, L _ { y } ) + \mathcal { L } _ { G A N } ^ { K } ( G _ { K }, D _ { K }, x ^ { * }, L _ { x } ). \quad \begin{pmatrix} \text{on} \, F \\ \text{Info} \\ \text{Pos} \end{pmatrix}$$

Keypoint Cycle-Consistency Loss. To better learn both keypoint cycles, we propose a keypoint cycle-consistency loss, which can be expressed as:

$$\cdot r \dashstr { r \dashstr { - } } \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \cdots \colon \\ \mathcal { L } _ { C Y C } ^ { K } ( G _ { K }, G _ { I }, x, y ^ { * }, L _ { x }, L _ { y } ) = & \inf _ { \inf _ { \mathcal { E } } } \\ \mathbb { E } _ { x, L _ { y } \sim p _ { \text{data} } ( x, L _ { y } ) } [ \| G _ { K } ( G _ { I } ( x, L _ { y } ) ) - L _ { y } \| _ { 1 } ] + \quad ( 1 1 ) \\ \mathbb { E } _ { y ^ { * }, L _ { x } \sim p _ { \text{data} } ( y ^ { * }, L _ { x } ) } [ \| G _ { K } ( G _ { I } ( y ^ { * }, L _ { x } ) ) - L _ { x } \| _ { 1 } ]. \\ \cdot \colon \colon \colon \colon \colon = 1 \, \detectors \colon \colon \text{in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the Main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main in the main.$$

We use L 1 distance in the keypoint cycle-consistency loss to compute pixelto-pixel difference between the generated keypoints L ∗ x , L ∗ y and the real keypoints L x , L y . During the training stage, the keypoint cycle-consistency loss can backpropagate errors from the keypoint generator to image generator, which facilitates the optimization of the image generator and thus improves the image generation.

## 3.4 Joint Optimization Objective

We also note that pixel loss [20, 39] can be used to reduce changes and constrain generators. Thus we adopt the image pixel loss between the real images y and the generated images y ∗ . We express this loss as:

$$\mathcal { L } _ { P I X E L } ^ { I } ( G _ { I }, x, L _ { y }, y ) = \mathbb { Z } _ { x, L _ { y }, y \sim \rho _ { \text{data} } ( x, L _ { y }, y ) } [ \| G _ { I } ( x, L _ { y } ) - y \| _ { 1 } ]. \quad \omega.$$

Weadopt L 1 distance as loss measurement in image pixel loss. Consequently, the complete objective loss is:

(12)

$$\dots \dots _ { r } \dots \dots \dots \dots \dots \dots \dots \dots \dots \dots \\ & \mathcal { L } ( G _ { I }, G _ { K }, D _ { I }, D _ { K } ) = \\ & \lambda _ { g a n } ^ { I } \ast \mathcal { L } _ { G A N } ^ { I } + \lambda _ { c y c } ^ { I } \ast \mathcal { L } _ { C Y C } ^ { I } + \lambda _ { p i x e l } ^ { I } \ast \mathcal { L } _ { P I X E L } ^ { I } + \quad ( 1 3 ) \quad \text{(13)} \quad \text{all t} \\ & \lambda _ { g a n } ^ { K } \ast \mathcal { L } _ { G A N } ^ { K } + \lambda _ { c y c } ^ { K } \ast \mathcal { L } _ { C Y C } ^ { K }, \\. \quad. \quad. r \quad. r \quad. r \quad. v \quad. w \quad. w \quad. \dots \. \.$$

where λ I дan , λ I cyc , λ I pixel , λ K дan and λ K cyc are parameters controlling the relative relation of objectives terms. We aim to solve:

$$G _ { I } ^ { * }, G _ { K } ^ { * } = \arg \min _ { G _ { I }, G _ { K } } \max _ { D _ { I }, D _ { K } } \mathcal { L } ( G _ { I }, G _ { K }, D _ { I }, D _ { K } ). \quad ( 1 4 ) \quad ^ { 1, 4 0 } ($$

## 3.5 Implementation Details

In this section, we introduce the detailed network implementation, the training strategy and the inference.

Network Architecture. For a fair comparison, we use the U-net architecture in PG 2 [20] as our generators. The encoder of generators is built with the basic Convolution-BatchNorm-LReLU layer. The decoder of generators is built with the basic Convolution-BatchNorm-ReLU layer. The leaky ReLUs in the encoder has a slope 0.2, while all ReLUs in the decoder are not leaky.

After the last layer, a Tanh function is used. We employ the PatchGAN discriminator [13, 60] as our discriminators D I and D K . The discriminators are built with the basic Convolution-BatchNorm-ReLU layer. All ReLUs are leaky, with slope 0.2. After the last layer, a convolution is applied to map it to a 1-D value, followed by a Sigmoid function.

Inference. At inference time, we follow the same settings of PG 2 [20] and PoseGAN [39] via inputting an image x and a target keypoint L y into the image generator G I , and then obtain the output target image. Similarly, the keypoint generator G K receives the image x as input and then outputs the keypoint of image x . We employ the same setting at both training and inference stage.

Training Strategy. We follow the standard optimization method from [11] to optimize the proposed C 2 GAN, i.e., we alternate between one gradient descent step on G I , D I , G K , and D K , respectively. The proposed C 2 GAN is trained end-to-end and can generate image and keypoint image simultaneously, then the generated keypoint will benefit the quality of the generated image. Moreover, in order to slow down the rate of discriminators D I , D K relative to generators G I , G K we divide the objectives by 2 while optimizing discriminators D I , D K . To enforce discriminators to remember what it has done wrong or right before, we use a history of generated images to update discriminators similar in [60]. Moreover, we employ OpenFace [1] and OpenPose [4] to extract keypoint images L x and L y on the Radboud Faces and Market-1501 datasets, respectively. Keypoint of Market-1501 dataset are represented as heatmaps similar as in PG 2 [20]. In contrast, we set the background of the heatmap to white color and the keypoint to black color on Radboud Faces dataset.

## 4 EXPERIMENTS

In this section, we first introduce the details of the datasets used in our experiments, and then we demonstrate the effectiveness of the proposed C 2 GAN and training strategy by presenting and analyzing qualitative and quantitative results.

## 4.1 Experimental Setup

Datasets. Weemploy two publicly datasets to validate the proposed C 2 GAN on two different tasks, including Radboud Faces dataset [19] for landmarkguided facial expression generation task, and Market-1501 dataset [58] for keypoint-guided person image generation task.

(ii) The Market-1501 dataset [58] is a more challenging person re-id dataset and we use it for the person keypoint and person image generation task. This dataset contains 32,668 images of 1,501 persons captured from six disjoint surveillance cameras. Persons vary in pose, illumination, viewpoint and background in this dataset, which makes the person image generation task more challenging. We follow the setup in PoseGAN [39]. For the training subset, we obtain 263,631 pairs, which is composed of two images of the same person but different poses. For testing subset, we randomly select 12,000 pairs. Note that no person overlapping between the training and testing subsets in this dataset.

(i) The Radboud Faces dataset [19] contains over 8,000 color face images collected from 67 subjects with eight different emotional expressions, i.e., anger, fear, disgust, sadness, happiness, surprise, neutral and contempt. It contains 1,005 images for each emotion and is captured from five cameras with different angles, and each subject is asked to show three different gaze directions. For each emotion, we randomly select 67% of images as training data and the rest 33% images as testing data. Different from StarGAN [7], all the images in our experiments are re-scaled to 256 × 256 × 3 without any pre-processing. For the landmark-guided facial expression generation task, we need pairs of images of the same face with two different expressions. We first remove those images in which the face is not detected correctly using the public OpenFace software [1], leading to 5,628 training image pairs and 1,407 testing image pair.

Figure 4: Qualitative results of ablation study on Radbound Faces dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-ba64323a-f590-44ca-b4d8-e9be0d1f049d.jpg)

Table 1: Quantitative results of ablation study on Radbound Faces dataset. For all metrics, higher is better.

| Baseline                     |   AMT |    PSNR |   SSIM |
|------------------------------|-------|---------|--------|
| C 2 GAN w/ I2I2I             |  25.3 | 21.203  | 0.8449 |
| C 2 GAN w/ I2I2I+K2G2K       |  28.2 | 20.8708 | 0.8419 |
| C 2 GAN w/ I2I2I+K2R2K       |  28.7 | 21.0156 | 0.8437 |
| C 2 GAN w/ I2I2I+K2G2K+K2R2K |  30.8 | 21.6262 | 0.854  |
| C 2 GAN w/ Single-Modal D    |  26.4 | 21.2794 | 0.8426 |
| C 2 GAN w/ Non-Sharing G     |  32.9 | 21.6353 | 0.8611 |

Parameter Setting. For both datasets, we do left-right flip for data augmentation similar in PG 2 [20]. For optimization, the proposed C 2 GAN is trained with a batch size of 16 on Radboud Faces dataset. For a fair comparison, all competing models were trained for 200 epochs on Radboud Faces dataset. We use the Adam optimizer [17] with the momentum terms β 1 = 0 5 and . β 2 = 0 999. The initial learning rate for Adam optimizer is 0.0002. For the . person image generation task, we train the model for 90 epochs with a smaller batch size 4.

For hyper-parameters setting, we fixed λ I дan and λ K дan to 1 and tune the rest using the grid search. We found that the weights of reconstruction losses (i.e., λ I cyc , λ I pixel , λ K cyc ) set between 10 and 100 yield good performance. Thus, λ I дan , λ I cyc , λ I pixel , λ K дan and λ K cyc in Eq. (13) are set to 1, 10, 10, 1 and 10, respectively. The proposed C 2 GAN is implemented using public deep learning framework PyTorch. To speed up the training and testing processes, we use an NVIDIA TITAN Xp GPU with 12G memory.

Moreover, we found that the keypoint generator cannot produce accurate keypoints in the early training stage since the image generator produces blurry images during this phase. Therefore, we employ a pre-trained OpenPose model [4], which replaces the keypoint generator to produce keypoints with location coordinates at the beginning of the training stage. We also minimize the distance between the generated keypoints (from the generated image) and the corresponding ground truth keypoints (from the ground truth image). Finally, we incorporate the mask loss proposed in PG 2 for person image generation task.

Evaluation Metric. We first adopt AMT perceptual studies to evaluate the quality of the generated images on both datasets similar to [20, 39]. To seek a quantitative measure that does not require human participation, Structural Similarity (SSIM) [53] and Peak Signal-to-Noise Ratio (PSNR) are employed to evaluate the quantitative quality of generated images on the Radboud Faces dataset.

For Market-1501 dataset on person image generation task, we follow [39] and use SSIM, Inception Score (IS) [37] and their corresponding masked versions mask-SSIM and mask-IS [20] as measurements.

Figure 5: Qualitative comparison with competitors on Radboud Faces dataset. From left to right: input image, ground truth target landmark, ground truth target image, GPGAN [8], PG 2 [20] and C GAN (ours). 2

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-e349688e-0039-440b-a8fc-7cd4bba6515b.jpg)

Table 2: Quantitative results of different methods on Radboud Faces dataset. For all metrics, higher is better.

| Model          | Publish   | AMT   |   SSIM |     PSNR |
|----------------|-----------|-------|--------|----------|
| GPGAN [8]      | ICPR 2018 | 0.3   | 0.8185 |  18.7211 |
| PG 2 [20]      | NIPS 2017 | 28.4  | 0.8462 |  20.1462 |
| C 2 GAN (Ours) | -         | 34.2  | 0.8618 |  21.9192 |
| Real Data      | -         | -     | 1      | inf      |

## 4.2 Model Analysis

We first investigate the effect of the combination of different individual generation cycles to demonstrate the importance of the proposed cycle-incycle network structure. Then the parameter-sharing strategy used in the generators for reducing the network capacity is evaluated, and finally the performance influence from the cross-modal discriminators is tested. All the comparison experiments are conducted via training the models for 50 epochs on Radbound Faces dataset. Fig. 4 shows examples of the qualitative results and Table 1 shows the quantitative results.

results and the removal of one of them degrades the generation performance, network I2I2I, demonstrating the effectiveness of the proposed C GAN. Cross-Modal Discriminator vs. Single-Modal Discriminator. We also evaluate the performance influence of the proposed cross-modal discriminator (C 2 GAN w/ I2I2I+K2G2K+K2R2K). Our baseline is the traditional single-modal discriminator (C 2 GAN w/ Single-Modal D). The single modalD means the discriminator receives only images as input, i.e., the real input images and the generated images. From Table 1, it is clear that the proposed cross-modal discriminator performs better than the single-modal discriminator on all evaluation metrics, meaning that the rich cross-modal information could help to learn better discriminator and thus facilitate the optimization of the generator.

Influence of Individual Generation Cycle. To evaluate the influence of individual generation cycle, we test with four different combinations of the cycles, i.e., I2I2I, I2I2I+K2G2K, I2I2I+K2R2K, and I2I2I+K2G2K+K2R2K. All four baselines use the same training strategies and hyper-parameters. As we can see in Table 1, I2I2I, K2G2K and K2R2K are all critical to our final demonstrating our initial intuition that by using cross-modal information in a joint generation framework and by making the cycles constraint on each other boost the final performance. I2I2I+K2G2K+K2R2K obtains the best performance, which is significantly better than the single cycle image 2

Parameter Sharing between Generators. The parameter sharing could remarkably reduce the parameters of the whole network. We further evaluate how parameter sharing would affect generation performance. We

Figure 6: More examples of the qualitative results of the proposed C GAN on the Market-1501 dataset. From left to right: input 2 image, ground truth target keypoint, ground truth target image and C GAN (ours). 2

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-0e44e161-4465-4fe1-a7af-a4f60ee693da.jpg)

test two different baselines: one is C 2 GAN w/ I2I2I+K2G2K+K2R2K, which shares the parameters of the two image generators G I and the two keypoint generators G K , respectively. While C 2 GAN w/ Non-Sharing G separately learns the four generators. We can observe from Table 1 that the non-sharing one achieves slightly better performance than sharing one. However, the number of parameters of non-sharing one is 217.6M, which doubles that of the sharing one. It means that the parameter sharing is a good strategy for balancing performance and overhead.

in Fig. 5. It is clear that GPGAN performs the worse among all the comparison models. While the results of PG 2 tend to be blurry. Compared with both GPGAN and PG 2 , the results of C 2 GAN are more smooth, sharper and contains more image details.

Existing competitors such as PG 2 [20] conduct the experiments on 256 × 256 resolution images. For a fair comparison with them, we also conduct experiments on this image resolution size but note that the proposed C 2 GAN can be applied for any size images with only small architecture modification.

## 4.3 Comparison against the State-of-the-Art

Competing Models. We consider several state-of-the-art keypoint-guided image generation models as our competitors, i.e., GPGAN [8], PG 2 [20], DPIG [21] and PoseGAN [39]. Note that for PG 2 , DPIG and PoseGAN, all of them need to use off-the-shelf keypoint detection models to extract the keypoints, and all of them are restricted on dataset sources or tasks where the keypoint information is available. Different from PG 2 , DPIG and PoseGAN, which focus on the person image generation task, the proposed C 2 GAN is a general model and learns image and keypoint generation simultaneously in a joint network. For a fair comparison, we implement all the models using the same setups as our approach.

Task 1: Landmark-Guided Facial Expression Generation. A qualitative comparison of different models on the Radboud Faces dataset is shown

Task 2: Keypoint-Guided Person Image Generation. Fig. 8 shows the results of PG 2 , PoseGAN and C 2 GAN on the Market-1501 dataset. As we can see, the proposed C 2 GAN is able to generate visually better images than PG 2 and PoseGAN. For the first row of results in Fig. 8, C 2 GAN can generate reasonable results while PG 2 cannot produce any meaningful content. PoseGAN can generate the person but cannot preserve the color information. For the second row of results in Fig. 8, both PoseGAN and PG 2 failed to generate the same child, while C 2 GAN can generate the same child with only a small part missing at the head. For the last row of results in Fig. 8, we can clearly observe the advantage of C 2 GAN as both PoseGAN and PG 2 failed to generate the hat. We also provide more qualitative results of C 2 GAN in Fig. 6. As we can see that the proposed C 2 GAN can generate photo-realistic images with convincing details. Moreover, the generated images are very close to the ground truths.

Figure 7: Visualization of keypoint generation on the Radboud Faces dataset.

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-940bfd91-f7a3-4332-9078-5d6ba76b3979.jpg)

Table 3: Quantitative results of different models on Market-1501 dataset. For all the metrics, higher is better.

| Model          | Publish   | AMT (R2G)   | AMT (G2R)   |   SSIM |    IS |   mask-SSIM |   mask-IS |
|----------------|-----------|-------------|-------------|--------|-------|-------------|-----------|
| PG 2 [20]      | NIPS 2017 | 11.2        | 5.5         |  0.253 | 3.46  |       0.792 |     3.435 |
| DPIG [21]      | CVPR 2018 | -           | -           |  0.099 | 3.483 |       0.614 |     3.491 |
| PoseGAN [39]   | CVPR 2018 | 22.7        | 50.2        |  0.29  | 3.185 |       0.805 |     3.502 |
| C 2 GAN (Ours) | -         | 23.2        | 46.7        |  0.282 | 3.349 |       0.811 |     3.51  |
| Real Data      | -         | -           | -           |  1     | 3.86  |       1     |     3.36  |

Figure 8: Qualitative comparison with PG 2 and PoseGAN on the Market-1501 dataset. From left to right: input image, ground truth target keypoint, ground truth target image, PoseGAN [39], PG 2 [20] and C GAN (ours). 2

![image](https://ac.marcodsn.me/arxiv-markdown-images/1908.00999v3-figure-f112ae4a-51cc-4be3-a030-251d4cdd590c.jpg)

Finally, we also note that GANs are difficult to train and easily have mode collapse. However, in keypoint-guided image generation tasks, avoiding mode collapse is not necessarily needed since if you input a person image and a target pose, the model tries to generate this particular person in this particular pose.

Quantitative Comparison of Both Tasks. We provide here quantitative results and analysis on both tasks. As shown in Table 2, C 2 GAN achieves the best performance on the Radboud Faces dataset with all the metrics for landmark-guided facial expression generation task. Moreover, we quantitatively compare the proposed C 2 GAN with PoseGAN, DPIG and PG 2 on keypoint-guided person image generation task in Table 3. We can observe that C 2 GAN obtains better performance than PG 2 and DPIG on all the evaluation metrics except for IS. Compared with PoseGAN, C 2 GAN yields very competitive performance. Specifically, we achieve better performance in terms of the AMT (R2G), IS, mask-SSIM and mask-IS metrics.

## 4.4 Visualization of Keypoint Generation

C 2 GAN is a cross-modal generation model and it is not only able to produce the target person but also able to produce the keypoint of the input image. Both generation tasks benefit from the improvement of each other in an end-to-end training fashion. We present examples of the keypoint generation results on Radboud Faces dataset in Fig. 7. The inputs are image x and keypoint L y and the outputs are image y ∗ and keypoint L ∗ x , other images and keypoints are given for comparison. As we can see that the generated keypoint L ∗ x is very close to the real keypoint L x , which verifies the effectiveness of the keypoint generator G k and the joint learning strategy.

## 5 CONCLUSION

In this paper, we propose a novel Cycle In Cycle Generative Adversarial Network (C 2 GAN) for keypoint-guide image generation task. C 2 GAN contains two different types of generators, i.e., keypoint-oriented generator and image-oriented generator. The image generator aims at reconstructing the target image based on a conditional image and the target keypoint, and the keypoint generator tries to generate the target keypoint and further provide cycle supervision to the image generator for generating more photorealistic images. Both generators are connected in a unified network and can be optimized in an end-to-end fashion. Both qualitative and quantitative experimental results on facial expression and person pose generation tasks demonstrate that our proposed framework is effective to generate high-quality images with convincing details.

## ACKNOWLEDGMENTS

We want to thank the NVIDIA Corporation for the donation of the TITAN Xp GPUs used in this work.

## REFERENCES

- [1] Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A general-purpose face recognition library with mobile applications . Technical Report. CMU-CS-16-118, CMU School of Computer Science.
- [3] Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018. Large scale gan training for high fidelity natural image synthesis. In ICLR .
- [2] Asha Anoosheh, Eirikur Agustsson, Radu Timofte, and Luc Van Gool. 2018. ComboGAN: Unrestrained Scalability for Image Domain Translation. In CVPR Workshop .
- [4] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime multiperson 2d pose estimation using part affinity fields. In CVPR .
- [6] Xinyuan Chen, Chang Xu, Xiaokang Yang, and Dacheng Tao. 2018. AttentionGAN for object transfiguration in wild images. In ECCV .
- [5] Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. 2018. Everybody dance now. In ECCV Workshop .
- [7] Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. StarGAN: Unified Generative Adversarial Networks for MultiDomain Image-to-Image Translation. In CVPR .
- [9] Brian Dolhansky and Cristian Canton Ferrer. 2018. Eye in-painting with exemplar generative adversarial networks. In CVPR .
- [8] Xing Di, Vishwanath A Sindagi, and Vishal M Patel. 2018. GP-GAN: gender preserving GAN for synthesizing faces from landmarks. In ICPR .
- [10] Bin Duan, Wei Wang, Hao Tang, Hugo Latapie, and Yan Yan. 2019. Cascade Attention Guided Residue Learning GAN for Cross-Modal Translation. arXiv preprint arXiv:1907.01826 (2019).
- [12] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. Improved training of wasserstein gans. In NIPS .
- [11] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In NIPS .
- [13] Phillip Isola, Junyan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-toimage translation with conditional adversarial networks. In CVPR .
- [15] Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In CVPR .
- [14] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive growing of gans for improved quality, stability, and variation. In ICLR .
- [16] Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jungkwon Lee, and Jiwon Kim. 2017. Learning to discover cross-domain relations with generative adversarial networks. In ICML .
- [18] Iryna Korshunova, Wenzhe Shi, Joni Dambre, and Lucas Theis. 2017. Fast faceswap using convolutional neural networks. In ICCV .
- [17] Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR .
- [19] Oliver Langner, Ron Dotsch, Gijsbert Bijlstra, Daniel HJ Wigboldus, Skyler T Hawk, and AD Van Knippenberg. 2010. Presentation and validation of the Radboud Faces Database. Taylor &amp; Francis Cognition and emotion (2010).
- [21] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. 2018. Disentangled person image generation. In CVPR .
- [20] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. 2017. Pose guided person image generation. In NIPS .
- [22] Shuang Ma, Jianlong Fu, Chang Wen Chen, and Tao Mei. 2018. DA-GAN: Instancelevel image translation by deep attention generative adversarial networks. In CVPR .
- [24] Michael Mathieu, Camille Couprie, and Yann LeCun. 2016. Deep multi-scale video prediction beyond mean square error. In ICLR .
- [23] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. 2015. Generating images from captions with attention. In ICLR .
- [25] Youssef Alami Mejjati, Christian Richardt, James Tompkin, Darren Cosker, and Kwang In Kim. 2018. Unsupervised attention-guided image-to-image translation. In NeurIPS .
- [27] Sangwoo Mo, Minsu Cho, and Jinwoo Shin. 2019. InstaGAN: Instance-aware Image-to-Image Translation. In ICLR .
- [26] Mehdi Mirza and Simon Osindero. 2014. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014).
- [28] Augustus Odena. 2016. Semi-supervised learning with generative adversarial networks. In ICML Workshop .
- [30] Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, and Jose M Álvarez. 2016. Invertible Conditional GANs for image editing. In NIPS Workshop .
- [29] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. 2019. Semantic image synthesis with spatially-adaptive normalization. In CVPR .
- [31] Fengchun Qiao, Naiming Yao, Zirui Jiao, Zhihao Li, Hui Chen, and Hongan Wang. 2018. Geometry-Contrastive Generative Adversarial Network for Facial Expression Synthesis. arXiv preprint arXiv:1802.01822 (2018).
- [32] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. 2016. Generative Adversarial Text-to-Image Synthesis. In ICML .
- [33] Scott Reed, Aäron van den Oord, Nal Kalchbrenner, Victor Bapst, Matt Botvinick, and Nando de Freitas. 2016. Generating interpretable images with controllable structure. Technical Report (2016).
- [35] Krishna Regmi and Ali Borji. 2018. Cross-view image synthesis using conditional gans. In CVPR .
- [34] Scott E Reed, Zeynep Akata, Santosh Mohan, Samuel Tenka, Bernt Schiele, and Honglak Lee. 2016. Learning what and where to draw. In NIPS .
- [36] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In MICCAI .
- [38] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. 2019. Animating arbitrary objects via deep motion transfer. In CVPR .
- [37] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In NIPS .
- [39] Aliaksandr Siarohin, Enver Sangineto, Stephane Lathuiliere, and Nicu Sebe. 2018. Deformable GANs for Pose-based Human Image Generation. In CVPR .
- [41] Qianru Sun, Liqian Ma, Seong Joon Oh, Luc Van Gool, Bernt Schiele, and Mario Fritz. 2018. Natural and effective obfuscation by head inpainting. In CVPR .
- [40] Lingxiao Song, Zhihe Lu, Ran He, Zhenan Sun, and Tieniu Tan. 2018. Geometry guided adversarial facial expression synthesis. In ACM MM .
- [42] Yaniv Taigman, Adam Polyak, and Lior Wolf. 2017. Unsupervised cross-domain image generation. In ICLR .
- [44] Hao Tang, Dan Xu, Nicu Sebe, Yanzhi Wang, Jason J Corso, and Yan Yan. 2019. Multi-channel attention selection gan with cascaded semantic guidance for crossview image translation. In CVPR .
- [43] Hao Tang, Wei Wang, Dan Xu, Yan Yan, and Nicu Sebe. 2018. GestureGAN for Hand Gesture-to-Gesture Translation in the Wild. In ACM MM .
- [45] Hao Tang, Dan Xu, Nicu Sebe, and Yan Yan. 2019. Attention-Guided Generative Adversarial Networks for Unsupervised Image-to-Image Translation. In IJCNN .
- [47] Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio.. In SSW .
- [46] Hao Tang, Dan Xu, Wei Wang, Yan Yan, and Nicu Sebe. 2018. Dual Generator Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. In ACCV .
- [48] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. 2016. Generating videos with scene dynamics. In NIPS .
- [50] Tingchun Wang, Mingyu Liu, Junyan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs. In CVPR .
- [49] Tingchun Wang, Mingyu Liu, Junyan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2018. Video-to-video synthesis. In NeurIPS .
- [51] Wei Wang, Xavier Alameda-Pineda, Dan Xu, Pascal Fua, Elisa Ricci, and Nicu Sebe. 2018. Every smile is unique: Landmark-guided diverse smile generation. In CVPR .
- [53] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE TIP 13, 4 (2004), 600-612.
- [52] Xiaolong Wang and Abhinav Gupta. 2016. Generative image modeling using style and structure adversarial networks. In ECCV .
- [54] Yichao Yan, Jingwei Xu, Bingbing Ni, Wendong Zhang, and Xiaokang Yang. 2017. Skeleton-aided Articulated Motion Generation. In ACM MM .
- [56] Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2017. SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient.. In AAAI .
- [55] Zili Yi, Hao Zhang, Ping Tan Gong, et al. 2017. DualGAN: Unsupervised Dual Learning for Image-to-Image Translation. In ICCV .
- [57] Jichao Zhang, Meng Sun, Jingjing Chen, Hao Tang, Yan Yan, Xueying Qin, and Nicu Sebe. 2019. GazeCorrection: Self-Guided Eye Manipulation in the wild using Self-Supervised Generative Adversarial Networks. arXiv preprint arXiv:1906.00805 (2019).
- [59] Shuchang Zhou, Taihong Xiao, Yi Yang, Dieqiao Feng, Qinyao He, and Weiran He. 2017. GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data. In BMVC .
- [58] Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. 2015. Scalable person re-identification: A benchmark. In ICCV .
- [60] Junyan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV .
Hao Tang Dan Xu Gaowen Liu Wei Wang Nicu Sebe Yan Yan
Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation
In this work, we propose a novel Cycle In Cycle Generative Adversarial
Network (C$^2$GAN) for the task of keypoint-guided image generation. The
proposed C$^2$GAN is a cross-modal framework exploring a joint exploitation of
the keypoint and the image data in an interactive manner. C$^2$GAN contains two
different types of generators, i.e., keypoint-oriented generator and
image-oriented generator. Both of them are mutually connected in an end-to-end
learnable fashion and explicitly form three cycled sub-networks, i.e., one
image generation cycle and two keypoint generation cycles. Each cycle not only
aims at reconstructing the input domain, and also produces useful output
involving in the generation of another cycle. By so doing, the cycles constrain
each other implicitly, which provides complementary information from the two
different modalities and brings extra supervision across cycles, thus
facilitating more robust optimization of the whole network. Extensive
experimental results on two publicly available datasets, i.e., Radboud Faces
and Market-1501, demonstrate that our approach is effective to generate more
photo-realistic images compared with state-of-the-art models.