  We propose pix2pix3D, a 3D-aware conditional generative model for
controllable photorealistic image synthesis. Given a 2D label map, such as a
segmentation or edge map, our model learns to synthesize a corresponding image
from different viewpoints. To enable explicit 3D user control, we extend
conditional generative models with neural radiance fields. Given
widely-available monocular images and label map pairs, our model learns to
assign a label to every 3D point in addition to color and density, which
enables it to render the image and pixel-aligned label map simultaneously.
Finally, we build an interactive system that allows users to edit the label map
from any viewpoint and generate outputs accordingly.

Computer Vision and Pattern Recognition;Graphics;Machine Learning