Multi-View Silhouette and Depth Decomposition for High Resolution 3D Object Representation
We consider the problem of scaling deep generative shape models to high-resolution. Drawing motivation from the canonical view representation of objects, we introduce a novel method for the fast up-sampling of 3D objects in voxel space through networks that perform super-resolution on the six orthographic depth projections. This allows us to generate high-resolution objects with more efficient scaling than methods which work directly in 3D. We decompose the problem of 2D depth super-resolution into silhouette and depth prediction to capture both structure and fine detail. This allows our method to generate sharp edges more easily than an individual network. We evaluate our work on multiple experiments concerning high-resolution 3D objects, and show our system is capable of accurately predicting novel objects at resolutions as large as 512×512×512 -the highest resolution reported for this task. We achieve state-of-the-art performance on 3D object reconstruction from RGB images on the ShapeNet dataset, and further demonstrate the first effective 3D super-resolution method.
Summary: This paper proposes a 3d super resolution method. It first projects the reconstructed 3D model to 6 low resolution orthographic depth maps. A mask network and a depth network are trained to up-sample the corresponding  depth maps to high resolution ones, and a 3D model carving strategy is applied to produce high resolution reconstructed result.   In their experiments, the reconstructed results are out perform the previous state-of-the-art pix2mesh algorithm by first do a low resolution reconstruction & then do super-resolution.  Novelty: The insight of the paper is leverage the difficulty of super-resolution in 3D to the field of well studied image super-resolution, so that the learning could be much easier.   Several questions seems not presented in the paper:  1: Is it still necessary to estimate the view point of the object in order to align the model to canonical views for up-sampling networks ?    2: For single image reconstruction task, the super-resolution in 3D is independent with image after low-res reconstruction. how is the result when the single image low resolution reconstruction fails, will the super-resolution enlarge the error ?  Maybe some failure case show be shown.  3: In L87, "[41] show the results up to 256^3". I think  this is because the dataset has ground truth at the resolution of 256^3, while the method has the capability to extend to 512^3 (as also shown in their paper), and they have the code released. The author should show a comparison on the same resolution (either visually or numerically).  Another problem of the paper is that the description is not good, e.g. L116: structure -> structural L125: passed -> passed in ?  the math symbols to represent different contents are somewhat sloppy ( what is the j, k in Eqn. (1) and (2), why depth map for high resolution is \hat{D}_H while for low resolution is C_L?  rather than  \hat{D}_L).   Experiments: The ablation study of how many canonical view (ODM) is needed w.r.t the final performance ? Since most object are symmetric in shapenet, maybe 3 is already good enough ?   How is the visual comparison between the proposed method with [41] ? It is important to have the intuition of the place of improvement for supporting the numbers.   Overall, although the method gives impressive results on reconstruction, I still have concern about the representation could be limited in real-world images since canonical views may not be easy to infer for non-rigid deformation and when occlusion happens. However, the paper proposes a effective method to alleviate the curse of dimension in 3D when the object is relative simple or symmetric.