Natural intelligent agents learn to perceive the three dimensional structure of the world without training on large datasets and are unlikely to have the precise equations of projective geometry hard-wired in the brain. Such skill would also be valuable to artificial systems in order to avoid the expensive collection of labeled datasets, as well as tedious tuning required by methods based on multi-view geometry. Inspired by natural agents, who interact with the environment via visual and haptic feedback, this paper explores a new approach to learning depth from images and very sparse depth measurements, just a few pixels per image. To learn from such extremely sparse supervision, we introduce an appropriate inductive bias by designing a specialized global-local network architecture. Experiments on several datasets show that the proposed model can learn monocular dense depth estimation when trained with very sparse ground truth, even a single pixel per image. Moreover, we find that the global parameters extracted by the network are predictive of the metric agent motion.
This paper presents a method for learning to predict a depth map given a pair of images. Training is done is an unsupervised way except for a small set of image locations for which the absolute 3D locations are known. The optical flow between the 2 images is computed automatically. A network is trained to predict the camera motion between the 2 images from the images and the optical flow. From this global motion, convolutional filters are predicted, in order to transform the optical flow into a depth map.
The use of image locations with known 3D locations is motivated in the paper to "simulate" the knowledge of depth coming from a haptic system.
I have several concerns about this paper.
* My main concern is that the paper compares against only a few papers (1 from 2014, 1 from  2016, and one more recent from 2019) while the literature is extremely vast, and visually, the results seem far from the state-of-the art. See for example:
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In CVPR, 2018.
Anurag Ranjan, Varun Jampani, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael J Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In CVPR, 2019.
Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using
direct methods. In CVPR, 2018.
In fact, these papers have simpler requirements as they do not need 2 images at run-time, nor 3d data at training time. I thought for a moment that the advantage of the paper was to be able to predict an absolute motion and an absolute depth (this is not impossible if the method is able to estimate the scale from known objects, as it is suggested from the introduction). This was however incorrect as the text says in the middle of Section 4.1 " we resolve the inherent scale ambiguity by normalizing the depth values such that the norm of the translation vector between the two views is equal to 1".
* The network used to predict optical flow was trained with a large amount of supervised data. As image matching is the most difficult task, it is difficult to claim that the method is unsupervised.
* The motivation for having sparse measurements is to "simulate" haptic. This is fine for me in principle, however haptic measurements would probably have large errors, while it seems that the experiments use ground truth values for these points.
One minor remark:  End of Section  4.4: What is a "179% error reduction"? How is computed the percentage?