  In this work, a system for creating a relightable 3D portrait of a human head
is presented. Our neural pipeline operates on a sequence of frames captured by
a smartphone camera with the flash blinking (flash-no flash sequence). A coarse
point cloud reconstructed via structure-from-motion software and multi-view
denoising is then used as a geometric proxy. Afterwards, a deep rendering
network is trained to regress dense albedo, normals, and environmental lighting
maps for arbitrary new viewpoints. Effectively, the proxy geometry and the
rendering network constitute a relightable 3D portrait model, that can be
synthesized from an arbitrary viewpoint and under arbitrary lighting, e.g.
directional light, point light, or an environment map. The model is fitted to
the sequence of frames with human face-specific priors that enforce the
plausibility of albedo-lighting decomposition and operates at the interactive
frame rate. We evaluate the performance of the method under varying lighting
conditions and at the extrapolated viewpoints and compare with existing
relighting methods.

Computer Vision and Pattern Recognition