Dual-Agent GANs for Photorealistic and Identity Preserving Profile Face Synthesis
Synthesizing realistic profile faces is promising for more efficiently training deep pose-invariant models for large-scale unconstrained face recognition, by populating samples with extreme poses and avoiding tedious annotations. However, learning from synthetic faces may not achieve the desired performance due to the discrepancy between distributions of the synthetic and real face images. To narrow this gap, we propose a Dual-Agent Generative Adversarial Network (DA-GAN) model, which can improve the realism of a face simulator's output using unlabeled real faces, while preserving the identity information during the realism refinement. The dual agents are specifically designed for distinguishing real v.s. fake and identities simultaneously. In particular, we employ an off-the-shelf 3D face model as a simulator to generate profile face images with varying poses. DA-GAN leverages a fully convolutional network as the generator to generate high-resolution images and an auto-encoder as the discriminator with the dual agents. Besides the novel architecture, we make several key modifications to the standard GAN to preserve pose and texture, preserve identity and stabilize training process: (i) a pose perception loss; (ii) an identity perception loss; (iii) an adversarial loss with a boundary equilibrium regularization term. Experimental results show that DA-GAN not only presents compelling perceptual results but also significantly outperforms state-of-the-arts on the large-scale and challenging NIST IJB-A unconstrained face recognition benchmark. In addition, the proposed DA-GAN is also promising as a new approach for solving generic transfer learning problems more effectively. DA-GAN is the foundation of our submissions to NIST IJB-A 2017 face recognition competitions, where we won the 1 st places on the tracks of verification and identification.
This work uses GANs to generate synthetic data to use for supervised training of facial recognition systems. More specifically, they use an image-to-image GAN to improve the  quality of faces generated by a face simulator. The simulator is able to produce a wider range of face poses for a given face, and the GAN is able to refine the simulators output such that it is more closely aligned with the true distribution of faces (i.e. improve the realism of the generated face) while maintaining the facial identity and pose the simulator outputted. They show that by fine tuning a facial recognition system on this additional synthetic data they are able to improve performance and outperform previous state of the art methods. 
Pros:
- This method is simple, apparently effective and is a nice use of GANs for a practical task. The paper is clearly written
Cons:
- My main concern with this paper is regarding the way in which the method is presented. The authors term their approach "Dual Agent GANS" and seem to claim a novel GAN architecture. However, it is not clear to me what aspect of their GAN is particularly new. The "dual agent"aspect of their GAN comes from the fact that they have a standard adversarial term (in their case the BE-GAN formulation) plus a cross entropy  term to ensure the facial identity is preserved. But previous work (e.g. InfoGAN, Auxiliary classifier GAN) have both also utilized a combination of "heads". So it seems odd to me that the authors are pushing this work as a new GAN architecture/method. I realize it's very trendy these days to come up with a slightly new GAN architecture and give it a new cool name, but this obfuscates the contributions. I think this is an interesting paper from perspective of using GANs in a data augmentation pipeline (and certainly their particular formulation is tailored to the task at hand) but I do not like that the authors appear to be claiming a new GAN method. 
- Since I think the main contribution of this paper is a data augmentation technique for facial recognition systems, it'd be good to see > 1 dataset explored. 
Some additional comments/questions:
- In eq. 8, do you mean to have a minus sign in the L_G term?
- What was the performance of the network you trained before fine tuning? i.e. how much improvement comes from this technique vs. different/better architectures/hyper-parameters/etc. compared to other methods