3D-Aware Scene Manipulation via Inverse Graphics
We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.
== Summary == This paper tackles the road scene manipulation problem by disentangling texture, semantic, and geometric information from a single image. A deep encoder-decoder network is introduced to learn such a disentangled representation. For each of the object instance, the method further infers the 3D object pose as its 3D-aware representation. Several image manipulation applications are demonstrated, thanks to a fully-disentangled 3D-aware representation. Experimental evaluations are conducted on KITTI and Cityscape benchmarks.   == Quality and Originality == Overall, this paper is quite interesting and experimental results are promising. However, reviewer does have a few concerns about the proposed 3D-aware model. Hopefully, these concerns can be addressed in the rebuttal.  == Object shape representation == Reviewer is a bit concerned as only one car model is used across all car instances (L169-L170). It does not sound like a principled approach for 3D disentangling in general. For example, the proposed approach is going to fail on object categories such as trucks, cyclists, or pedestrians.   == Object occlusion == Reviewer is also concerned whether the proposed 3D-aware geometric inference method is robust to partial occlusions. For example, it might not be a good strategy to treat each car independently if one is occluded by the other. In some cases, the context information could help when there is partial occlusion. However, reviewer is not sure whether the current architecture (Figure 2: looks like the three streams are independent) addresses the issue mentioned here.   == Indoor vs. Road Scene == As it is not shown in the main paper, reviewer would like to know the performance in indoor scene. This is a very critical point, as the title of the submission is about scene manipulation instead of road scene manipulation.  == Missing references == References are incomplete regarding the interpretable 2D/3D representation in deep encoder-decoder networks. -- Attribute2Image: Conditional Image Generation from Visual Attributes, Yan et al. In ECCV 2016. -- Neural Face Editing with Intrinsic Image Disentangling, Shu et al. In CVPR 2017. -- Material Editing Using a Physically Based Rendering Network, Liu et al. In ICCV 2017. -- MoFA: Model-based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction, Tewari et al. In ICCV 2017. -- Self-Supervised Intrinsic Image Decomposition, Janner et al. In NIPS 2017.