Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
Predicting depth is an essential component in understanding the 3D geometry of a scene. While for stereo images local correspondence suffices for estimation, finding depth relations from a single image is less straightforward, requiring integration of both global and local information from various cues. Moreover, the task is inherently ambiguous, with a large source of uncertainty coming from the overall scale. In this paper, we present a new method that addresses this task by employing two deep network stacks: one that makes a coarse global prediction based on the entire image, and another that refines this prediction locally. We also apply a scale-invariant error to help measure depth relations rather than scale. By leveraging the raw datasets as large sources of training data, our method achieves state-of-the-art results on both NYU Depth and KITTI, and matches detailed depth boundaries without the need for superpixelation.
The authors present a new method for estimating the depth map of a scene using a single image. They use two CNNs: the first outputs a coarse depth prediction based on the entire image. This coarse depth prediction is used as the input, together with the original image, to the second CNN which refines the depth prediction locally. They also present and apply a scale-invariant error measure for the depth prediction. They demonstrate state of the art results on NYU Depth and KITTI datasets.The paper is well written and clear, I believe researchers who would like to reproduce the results could do that using the details found in the paper. I also find that the problem they solve - estimating the depth of a scene using a single image, is relevant and interesting to the NIPS community.However, much of the significance of the paper and the interest it raises is related to the two CNNs it uses - global and local. I would like to see better analysis  of the importance of this two networks. It seems that quantitatively there isn't much of a difference to when using only the coarse CNN and both CNNs. How would the results look like if we would feed the second CNN with some other depth prediction which is not "coarse" (e.g. the output of Make3D), or if the first CNN will also be more local?In the end of the first paragraph of 3.1.1 the authors explain the importance of global cues for depth estimation, such as vanishing points and room alignment. Is there a way to understand the patterns learned as such cues?The fact that the choice of the lambda parameter of the loss doesn't give numeric gain but changes the quality of the prediction is a bit troubling (as the use of the second network)  and suggests that the error measure used is not the most suited for this task.I must say that when examining qualitatively the results presented both in the paper and in the supplementary material it seems that: (a) the depth estimate is still quite far from the ground truth. (b) on the KITTI dataset, it doesn't seem to me that the suggested architecture is better than Move3D.I would like to see the running time and the computational resources needed for training and testing.Minor comment:Can it be that you forgot a factor 2 when moving from eq. 2 to eq. 3?