RenderNet: A deep convolutional network for differentiable rendering from 3D shapes
Traditional computer graphics rendering pipelines are designed for procedurally generating 2D images from 3D shapes with high performance. The nondifferentiability due to discrete operations (such as visibility computation) makes it hard to explicitly correlate rendering parameters and the resulting image, posing a significant challenge for inverse rendering tasks. Recent work on differentiable rendering achieves differentiability either by designing surrogate gradients for non-differentiable operations or via an approximate but differentiable renderer. These methods, however, are still limited when it comes to handling occlusion, and restricted to particular rendering effects. We present RenderNet, a differentiable rendering convolutional network with a novel projection unit that can render 2D images from 3D shapes. Spatial occlusion and shading calculation are automatically encoded in the network. Our experiments show that RenderNet can successfully learn to implement different shaders, and can be used in inverse rendering tasks to estimate shape, pose, lighting and texture from a single image.
The paper proposes an end-to-end differentiable renderer which takes a voxelized shape as input and outputs the rendered image.  Pros: The idea is clean and simple. Compared to previous differentiable renderers, it can condition on lights and textures, and is extensible to inverse rendering tasks. When applying on inverse rendering tasks on human faces, it can generate images of better quality.  Cons: 1. The projection unit and 2D convolution (the rendering part) may not be necessary to learn. For example, in Phong shading, the normals can be computed using gradients of the depth map, and the dot operation of the lighting direction and normals can also be differentiable. In Barron & Malik 2015 they already have such differentiable renderer with lighting represented by spherical harmonics. It is also unclear to me why the visibility of objects need to be learned with the projection network. Don't you just need to pick the voxel closest to the camera to know visibility? 2. In Section 4.2, it may not be fair since the other networks need to learn the 3D transformations, while in RenderNet, the 3D transformations are explicitly done. 3. In figure 2, marching cubes is an algorithm to generate a mesh from 3D voxels. I am not sure why it is compared with renderers. 4. It would be more convincing if the reconstructed 3D shape in section 4.3 can be shown in a figure. 5. Lack of quantitative evaluation in section 4.3.  I also have some questions about the technical details: 1. I am confused about the output of the RenderNet. In line 136 the paper mentions the output is the shading, but in Figure 1 the output seems to be the Normal Map which is converted to shading. Line 137 also mentions that normal is an optional output. If you directly output shading, do you mean the Phong shading can be learned by the network itself without the need to explicitly specifying S = max(0, l*n + a) as in line 175? In other styles of rendering, do you also directly output the final shading? How do you use the input light position in those cases?  Please address the above concerns.