  In this paper, we propose a geometric neural network with edge-aware
refinement (GeoNet++) to jointly predict both depth and surface normal maps
from a single image. Building on top of two-stream CNNs, GeoNet++ captures the
geometric relationships between depth and surface normals with the proposed
depth-to-normal and normal-to-depth modules. In particular, the
"depth-to-normal" module exploits the least square solution of estimating
surface normals from depth to improve their quality, while the
"normal-to-depth" module refines the depth map based on the constraints on
surface normals through kernel regression. Boundary information is exploited
via an edge-aware refinement module. GeoNet++ effectively predicts depth and
surface normals with strong 3D consistency and sharp boundaries resulting in
better reconstructed 3D scenes. Note that GeoNet++ is generic and can be used
in other depth/normal prediction frameworks to improve the quality of 3D
reconstruction and pixel-wise accuracy of depth and surface normals.
Furthermore, we propose a new 3D geometric metric (3DGM) for evaluating depth
prediction in 3D. In contrast to current metrics that focus on evaluating
pixel-wise error/accuracy, 3DGM measures whether the predicted depth can
reconstruct high-quality 3D surface normals. This is a more natural metric for
many 3D application domains. Our experiments on NYUD-V2 and KITTI datasets
verify that GeoNet++ produces fine boundary details, and the predicted depth
can be used to reconstruct high-quality 3D surfaces. Code has been made
publicly available.

Computer Vision and Pattern Recognition