SURGE: Surface Regularized Geometry Estimation from a Single Image
This paper introduces an approach to regularize 2.5D surface normal and depth predictions at each pixel given a single input image. The approach infers and reasons about the underlying 3D planar surfaces depicted in the image to snap predicted normals and depths to inferred planar surfaces, all while maintaining fine detail within objects. Our approach comprises two components: (i) a fourstream convolutional neural network (CNN) where depths, surface normals, and likelihoods of planar region and planar boundary are predicted at each pixel, followed by (ii) a dense conditional random field (DCRF) that integrates the four predictions such that the normals and depths are compatible with each other and regularized by the planar region and planar boundary information. The DCRF is formulated such that gradients can be passed to the surface normal and depth CNNs via backpropagation. In addition, we propose new planar-wise metrics to evaluate geometry consistency within planar surfaces, which are more tightly related to dependent 3D editing applications. We show that our regularization yields a 30% relative improvement in planar consistency on the NYU v2 dataset [24].
See below.
The paper proposes a method for recovering scene geometry from a single RGB image. This method uses a dense CRF with terms that enforce consistency between point-wise depth and normal estimates, using regularizers based on classification of planarity and presence of depth boundaries. Each of these estimates (depth, normal, planarity, edges) comes from a separate network proposed for each task in prior work. Experimental evaluation show modest improvement over baseline depth and normal prediction models.  In addition to the geometry-terms in proposed DCRF-based model, the paper's contributions include using multiple passes through the depth and normal networks with dropout to derive 'confidence' values of these metrics, and joint training to fine tune the depth and normal networks.   While significantly engineered for its specific application domain, the paper does demonstrate a successful example of inference with a regularized objective, where different terms are predicted from trained neural networks. I believe this will be of interest to many in the NIPS audience.  However, the overall presentation of the method---including some of the motivation and evaluation---leaves some questions un-answered:  - I have to say the benefits from joint training don't appear to be significant---could this improvement be explained by essentially additional training iterations ? Perhaps, a better test would've been to train these networks from scratch, or perhaps further train the depth and normal networks for an equal number of further iterations on the same training set (but with direct losses instead of post-DCRF losses).  - The fact that Eigen-VGG(JT) does better than the vanilla version is a little surprising---the joint training optimizes each network to perform better within the DCRF with complimentary information from the others. Why would it improve stand-alone performance ? Surely the direct loss used in the original training of each of these networks is the right choice for stand-alone performance. This again makes me believe that the improvement seen due to joint-training is because of the extra training iterations, and not because of the "joint" loss.  - Why were confidences not used during joint training ? One could "tie together" multiple forward passes for the same example, and use that to derive confidence values, and backprop through these. Is it that the extra forward passes weren't worth the slower training time ? It is also worth reporting results for DCRF (JT)---i.e., DCRF results without confidence values, but with joint training---since this corresponds to using the network in the same regime for which it was trained.  - While most of the terms in the DCRF objective "make sense", they are hand-crafted and involve making a lot of design decisions. Given the small size of the NYUv2 test set (and that its GT data is available), it is worth mentioning whether all these design decisions were made on the val set. Along similar lines, how did authors decide on the parametric form for dropout-variance to confidence ? Was this also determined on the val set from [9] ?  - The work of [28] is perhaps the closest to the proposed method, since it also uses a dense CRF with joint terms for depth and edges. I think the paper could use a more thorough discussion of it. [28]'s results are worse than those reported in this paper, but that may be because of the VGG-trained [6] baseline network used by this paper. In fact, [28] appears to show a more significant relative improvement between its depth-only and depth+edge CRFs (than this paper's DCRF-conf(JT) vs [6]). A fair question to ask is how would [28]'s "HCRF" do with [6]'s depth network and VGG-pretraining.  - The paper could do with a formal definition of 'planarity' means in the P0 output---specifically, at what resolution / region-size is point-wise planarity defined (in the limit, every infinitesimal surface point is a plane!) ? Is a planar region defined as one with some minimum number of contiguous pixels in which normals are constant ?  - The baseline errors for [6] appear to be different from those reported in [6] itself. Could the authors comment ? (the biggest difference appears to be in the RMSE errors---are they computing means of mean(per-image-RMSE), or sqr-root(MSE across all images). I believe the latter is the convention used to report results on NYUv2.  ===Post-rebuttal The authors have largely addressed the questions in my review. I'd say the paper just about meets the threshold for publication at NIPS (with the promised additional discussions for clarity).  I'd also recommend that authors consider doing joint training with the RAW distribution of the NYUv2 set (rather than just the smaller official release). I think the current experiments might be underselling the utility of the method because of the limited training data.