  We propose a novel framework for creating large-scale photorealistic datasets
of indoor scenes, with ground truth geometry, material, lighting and semantics.
Our goal is to make the dataset creation process widely accessible,
transforming scans into photorealistic datasets with high-quality ground truth
for appearance, layout, semantic labels, high quality spatially-varying BRDF
and complex lighting, including direct, indirect and visibility components.
This enables important applications in inverse rendering, scene understanding
and robotics. We show that deep networks trained on the proposed dataset
achieve competitive performance for shape, material and lighting estimation on
real images, enabling photorealistic augmented reality applications, such as
object insertion and material editing. We also show our semantic labels may be
used for segmentation and multi-task learning. Finally, we demonstrate that our
framework may also be integrated with physics engines, to create virtual
robotics environments with unique ground truth such as friction coefficients
and correspondence to real scenes. The dataset and all the tools to create such
datasets will be made publicly available.

Computer Vision and Pattern Recognition