Single-Image Depth Perception in the Wild
This paper studies single-image depth perception in the wild, i.e., recovering depth from a single image taken in unconstrained settings. We introduce a new dataset "Depth in the Wild" consisting of images in the wild annotated with relative depth between pairs of random points. We also propose a new algorithm that learns to estimate metric depth using annotations of relative depth. Compared to the state of the art, our algorithm is simpler and performs better. Experiments show that our algorithm, combined with existing RGB-D data and our new relative depth annotations, significantly improves single-image depth perception in the wild. Figure 1: We crowdsource annotations of relative depth and train a deep network to recover depth from a single image taken in unconstrained settings ("in the wild").
Is it possible to collect ground-truth depth for images in the wild? Using depth sensors in unconstrained settings is not yet feasible. Crowdsourcing seems viable, but humans are not good at estimating metric depth, or 3D metric structure in general [13]. In fact, metric depth from a single image is fundamentally ambiguous: a tree behind a house can be slightly bigger but further away, or slightly smaller but closer-the absolute depth difference between the house and the tree cannot be uniquely determined. Furthermore, even in cases where humans can estimate metric depth, it is unclear how to elicit the values from them.
But humans are better at judging relative depth [13]: "Is point A closer than point B?" is often a much easier question for humans. Recent work by Zoran et al. [14] shows that it is possible to learn to estimate metric depth using only annotations of relative depth. Although such metric depth estimates are only accurate up to monotonic transformations, they may well be sufficiently useful for high-level tasks, especially for occlusion reasoning. The seminal results by Zoran et al. point to two fronts for further progress: (1) collecting a large amount of relative depth annotations for images in the wild and (2) improving the algorithms that learn from annotations of relative depth.
In this paper, we make contributions on both fronts. Our first contribution is a new dataset called "Depth in the Wild" (DIW). It consists of 495K diverse images, each annotated with randomly sampled points and their relative depth. We sample one pair of points per image to minimize the redundancy of annotation 1 . To the best of our knowledge this is the first large-scale dataset consisting of images in the wild with relative depth annotations. We demonstrate that this dataset can be used as an evaluation benchmark as well as a training resource 2 .
Our second contribution is a new algorithm for learning to estimate metric depth using only annotations of relative depth. Our algorithm not only significantly outperforms that of Zoran et al. [14], but is also simpler. The algorithm of Zoran et al. [14] first learns a classifier to predict the ordinal relation between two points in an image. Given a new image, this classifier is repeatedly applied to predict the ordinal relations between a sparse set of point pairs (mostly between the centers of neighboring superpixels). The algorithm then reconstructs depth from the predicted ordinal relations by solving a constrained quadratic optimization that enforces additional smoothness constraints and reconciles potentially inconsistent ordinal relations. Finally, the algorithm estimates depth for all pixels assuming a constant depth within each superpixel.
In contrast, our algorithm consists of a single deep network that directly predicts pixel-wise depth (Fig. 1). The network takes an entire image as input, consists of off-the-shelf components, and can be trained entirely with annotations of relative depth. The novelty of our approach lies in the combination of two ingredients: (1) a multi-scale deep network that produces pixel-wise prediction of metric depth and (2) a loss function using relative depth. Experiments show that our method produces pixel-wise depth that is more accurately ordered, outperforming not only the method by Zoran et al. [14] but also the state-of-the-art image-to-depth system by Eigen et al. [8] trained with ground-truth metric depth. Furthermore, combing our new algorithm, our new dataset, and existing RGB-D data significantly improves single-image depth estimation in the wild.
In this paper, the authors collected a new dataset for training 3D reconstruction from a single image. Unlike previous datasets that recorded the true 3D structure of the scenes, this dataset only records the relative depth of a pair of points in each image. It is because that compared to the annotating absolute depth of the environment, it is easier for human beings to judge the relative depths of two certain points in the scene. In addition, the authors designed a new method that used annotations of relative depth to learn a model for single-image 3D reconstruction. 
First, since the new dataset is one of the key contributions of this paper, I suggested the authors to provide more statistics of the dataset. For example, among the 495K images in the dataset, how many images describe large scenes? How many images focus on certain small objects? Among all the scene images, how many images describe indoor scenes, outdoor urban scenes, and outdoor wild scenes? Among all the object images, how many images describe man-made objects and animals? Based on such statistics, we can evaluate the generality of the new dataset.  Second, the new dataset can be only applied to the methods that can learn from ordinal relations between two points in an image. Thus, most existing methods cannot use this dataset.  Third, I do not understand why the authors only label a single pair of points in each image. I admit that labeling two pairs of points in two different images may provide more information than labeling two pairs of points in a single image, because the two pairs of points in a single image may share similar relative depth. However, there also are some advantages for annotating more pairs of points in a single image. 1) Labeling more points may greatly decrease the structural uncertainty of the image, thus providing more reliable training images. It is possible that a small number of reliable training images may contribute more to model learning than a large number of unreliable training images. 2) Labeling more points in a single image may make this dataset applicable to more methods, because only using one pair of annotations in an image is a very strict constraint.  Fourth, 50% of the point pairs for annotation were randomly selected, and the other 50% of point pairs had symmetric shapes. Such a design for point annotation is quite arbitrary. I suggest the authors to design a certain loss to achieve an active framework for point annotation, which uses the loss to identify the most informative point pairs in an image. This will be a more efficient way for dataset construction.  Fifth, the proposed method for learning from ordinal point relations has a minor novelty.