Learning a High Fidelity Pose Invariant Model for High-resolution Face Frontalization
Face frontalization refers to the process of synthesizing the frontal view of a face from a given profile. Due to self-occlusion and appearance distortion in the wild, it is extremely challenging to recover faithful results and preserve texture details in a high-resolution. This paper proposes a High Fidelity Pose Invariant Model (HF-PIM) to produce photographic and identity-preserving results. HF-PIM frontalizes the profiles through a novel texture warping procedure and leverages a dense correspondence field to bind the 2D and 3D surface spaces. We decompose the prerequisite of warping into dense correspondence field estimation and facial texture map recovering, which are both well addressed by deep networks. Different from those reconstruction methods relying on 3D data, we also propose Adversarial Residual Dictionary Learning (ARDL) to supervise facial texture map recovering with only monocular images. Exhaustive experiments on both controlled and uncontrolled environments demonstrate that the proposed method not only boosts the performance of pose-invariant face recognition but also dramatically improves high-resolution frontalization appearances.
The paper proposes a face frontalization method based on a combination of existing methods of dense correspondence-based texture estimation and warping. An adversarial network was also trained in assisting the improve the texture map learning. The proposed method was evaluated via face recognition and verification tasks.  Face frontalization is important specifically for applications in the wild with no pose and other conditional constraints. The topic is relevant to the NIPS audience. The paper reads good and is easy to follow. However, there are many weaknesses in the paper. I will list them as follows.  Major comments: ===============  - Since face recognition/verification methods are already performing well, the great motivation for face frontalization is for applications in the wild and difficult conditions such as surveillance images where pose, resolution, lighting conditions, etc… vary wildly. To this effect, the paper lacks sufficient motivation for these applications.  - The major drawback is the method is a collection of many existing methods and as such it is hard to draw the major technical contribution of the paper. Although a list of contributions was provided at the end of the introduction none of them are convincing enough to set this paper aside technically. Most of the techniques combined are standard existing methods and/or models. If the combination of these existing methods was carefully analyzed and there were convincing results, it could be a good application paper. But, read on below for the remaining issues I see to consider it as application paper.  - The loss functions are all standard L1 and L2 losses with the exception of the adversarial loss which is also a standard in training GANs. The rationale for the formulation of these losses is little or nonexistent.  - The results for face verification/recognition were mostly less than 1% and even outperformed by as much as 7% on pose angles 75 and 90 (see Table 1). The exception dataset evaluated is IJB-A, in which the proposed model performed by as much as 2% and that is not surprising given IJB-A is collected in well constrained conditions unlike LFW. These points are not discussed really well.  - The visual qualities of the generated images also has a significant flaw in which it tends to produce a more warped bulged regions (see Fig. 3) than the normal side. Although, the identity preservation is better than other methods, the distortions are significant. This lack of symmetry is interesting given the dense correspondence is estimated.   - Moreover, the lack of ablation analysis (in the main paper) makes it very difficult to pinpoint from which component the small performance gain is coming from.  - In conclusion, due to the collection of so many existing methods to constitute the proposed methods and its lack of convincing results, the computational implications do not seem warranted.  Minor comments: ===============  - Minor grammatical issue need to be checked here and there.  - The use of the symbol \hat for the ground truth is technically not appealing. Ground truth variables are usually represented by normal symbols while estimated variables are represented with \hat. This needs to be corrected throughout for clarity.