The present invention provides a novel approach to simultaneously extracting the 3D shape of the face and the semantically consistent 2D alignment using a 3D Spatial Transformer Network (3DSTN) to model both the camera projection matrix and the warping parameters of a 3D model. By utilizing a generic 3D model and a thin plate spline (TPS) warping function, subject-specific 3D shapes are able to be generated without the need for a large 3D shape basis.
3D spatial transformer network