FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification
Person re-identification (reID) is an important task that requires to retrieve a person's images from an image dataset, given one image of the person of interest. For learning robust person features, the pose variation of person images is one of the key challenges. Existing works targeting the problem either perform human alignment, or learn human-region-based representations. Extra pose information and computational cost is generally required for inference. To solve this issue, a Feature Distilling Generative Adversarial Network (FD-GAN) is proposed for learning identity-related and pose-unrelated representations. It is a novel framework based on a Siamese structure with multiple novel discriminators on human poses and identities. In addition to the discriminators, a novel same-pose loss is also integrated, which requires appearance of a same person's generated images to be similar. After learning pose-unrelated person features with pose guidance, no auxiliary pose information and additional computational cost is required during testing. Our proposed FD-GAN achieves state-of-the-art performance on three person reID datasets, which demonstrates that the effectiveness and robust feature distilling capability of the proposed FD-GAN. ‡ ‡
: The image encoder in the FD-GAN is trained to learn robust identity-related and poseunrelated representations with assistance of pose-guided image generator and discriminators. During inference, it does not need pose information and additional computational cost.
Generative adversarial network (GAN) is gaining increasing attention for image generation. Recently, some works exploited GANs' potential on aiding current person reID algorithms. Zheng et al. [3] proposed a semi-supervised structure which learns generative images with label smoothing regularization for outliers (LSRO) regularization. PTGAN [4] was proposed to bridge the domain gap between separate datasets. In addition to image synthesis, GAN can be used for representation learning as well. In this work, we propose a novel identity-related representation learning framework for robust person re-identification.
The proposed Feature Distilling Generative Adversarial Network (FD-GAN) maintains identity feature consistency under pose variation without increasing the complexity of inference (illustrated in Figure 1). It adopts a Siamese structure for feature learning. Each of the branch consists of an image encoder and an image generator. The image encoder embeds person visual features given the input images. The image generator generates new person images conditioned on the pose information and the input person features by the encoder. Multiple discriminators are integrated in the framework to distinguish inter-branch and intra-branch relations between generated images by the two branches.
The proposed identity discriminator, the pose discriminator, and the verification classifier together with a reconstruction loss and a novel same-pose loss jointly regularizes the feature learning process for achieving robust person reID. With the adversarial losses, identity-irrelevant information, such as pose and background appearance, in the input image is mitigated from the visual features by the image encoder. More importantly, during inference, additional pose information is no longer needed and saves additional computational cost. Our method outperforms previous works in three widely-used reID datasets, i.e. Market-1501 [5], CUHK03 [6] and DukeMTMC-reID [7] datasets.
Overall, this paper has the following contributions. 1) We propose a novel framework, FD-GAN, to learn identity-related and pose-unrelated representations for person re-identification with posevariation. Unlike existing alignment or region-based learning methods, our framework does not require extra auxiliary pose information or increase the computational complexity during inference. 2) Although person image generation is an auxiliary task for our framework, the generated person images by our proposed method show better quality than existing specific person-generation methods.
3) The proposed FD-GAN achieves state-of-the-art re-identification performance on Market-1501 [5], CUHK03 [6], and DukeMTMC-reID [7] datasets.
This paper describes a GAN approach to addressing a common and important problem in person re-identification: inter- and intra-view pose variation. The technique, in extreme synthesis, uses a generative model to implicitly marginalize away pose- and background-dependent information in the feature representaiton to distill a representation that is invariant to both, but still discriminative for person identities. Pose is represented as the spatial configuration of landmarks, and during training person images conditioned on a randomly selected pose are generated from image encodings. These generated images are fed to multiple adversarial discriminators that determine if the generated image is real/false, if the pose in a real/fake image is accurate, and if two feature embeddings correspond to the same person. Experimental results are given on multiple, important benchmark datasets and show significant improvement over the state-of-the-art.   Clarity, quality, and reproducibility: The clarity of exposition is quite good. The authors do a good job of describing a complicated combination of modules and losses, with in my opinion the right degree of precision and level of detail. There are some minor typos and grammar errors throughout, but not to distraction. The quality of the technical exposition and experimental evaluation is similarly high, and I appreciated how each component and loss term is methodically and incrementally presented in section 3.3.  I feel like the results of the paper would be challenging to reproduce, however this is mainly due to the large number of moving parts and not to lack of clarity or missing details.  On a more critical note, and related to reproducibility, generative adversarial architectures with just a *single* adversarial loss are notoriously unstable during training. The proposed system has five terms in the loss, and three phases of (what appears to be) carefully crafted training schedule. Was this training pipeline, with its myriad hyperparameters, derived on a validation re-identification dataset?  More insight into the behavior of the system at training time would be a useful contribution. Also, some ablation analysis showing the incremental contributions each loss would lend credibility to the need for each of the adversarial and identity modules and losses.  Novelty: This work uses the idea of generative adversarial models and applies it in a novel and clever way to learn identity discriminative feature representations that are robust to pose and background variations -- probably the two most significant challenges in person re-identification. This type of adversarial invariance learning has not been applied in person re-identification, and I am unaware of similar approaches in other applications (other than DR-GAN for pose-invariant face recognition, included in the comparative evaluation). The authors do an excellent job of positioning their work in the context of the state-of-the-art and analyzing the advantages of their approach.  Significance and fit with NIPS: The experimental results on multiple, modern, and large scale re-identification benchmarks show a significant improvement over the state-of-the-art. Though the proposed system is based on multiple known techniques, the combination is novel and well thought-out. This paper is fundamentally an architecture engineering paper, but an extremely well-executed one. It is likely to have significant impact in the person re-identification community, and could potentially contribute insights to adversarial invariant feature learning in the broader NIPS world.  Some specific comments and questions:   1. Figures 2 and 3 are gorgeous and do a good job of explaining the     complexities of the components. However their size has been     reduced to the limits of printed legibility (likely due to space).  Line 19: widely --> wide Line 103: regularizes --> regularize Line 113: same person's another image --> another image of the same person Line 210: integratie --> integrate  POST REBUTTAL: The rebuttal addressed my main concerns (and I was already quite positive). The clarifications about training and crossvalidation, plus the additional ablation table, make the paper even stronger.