DPSNET: END-TO-END DEEP PLANE SWEEP STEREO
Multiview stereo aims to reconstruct scene depth from images acquired by a camera under arbitrary motion. Recent methods address this problem through deep learning, which can utilize semantic cues to deal with challenges such as textureless and reflective regions. In this paper, we present a convolutional neural network called DPSNet (Deep Plane Sweep Network) whose design is inspired by best practices of traditional geometry-based approaches for dense depth reconstruction. Rather than directly estimating depth and/or optical flow correspondence from image pairs as done in many previous deep learning methods, DPSNet takes a plane sweep approach that involves building a cost volume from deep features using the plane sweep algorithm, regularizing the cost volume via a contextaware cost aggregation, and regressing the dense depth map from the cost volume. The cost volume is constructed using a differentiable warping process that allows for end-to-end training of the network. Through the effective incorporation of conventional multiview stereo concepts within a deep learning framework, DPSNet achieves state-of-the-art reconstruction results on a variety of challenging datasets.
The paper describes a method for learning a deep neural network for multi-view stereo. The overall network includes feature-extraction layers applied to all images, followed by a spatial-transformer network (which is differentiable, but with no learnable parameters) that is applied to warp these features from every matching image to the reference image's co-ordinate frame for a series of candidate depth planes, followed by concatenation of the reference and match image features and 3D convolution layers to form a cost volume. The cost volumes of different pairs are averaged, and additional layers are used to refine this cost volume while relying on the reference image's RGB features, followed by soft-max and an expectation over depth values to output the final depth at each pixel. The entire network is trained end-to-end and experiments show that it outperforms state-of-the-art methods for MVS by a significant margin on a number of datasets.
Overall, I have a positive view of the paper and believe it should be accepted to ICLR. However, I would like the authors to address the following issues:
- While the proposed network is complex, I do believe the description of the architecture could be a little better. It would be good to clarify that i indexes view (and N is the total number of views), and provide a few more definitions for the terms in equation (2): namely, are R and t the extrinsics of the reference camera or the i^th camera, etc. The overall approach is clear (for each plane, the method maps features from the paired camera to the reference camera  assuming all points in the the world lie on that plane), but it would be good to clarify the specifics. It might also be useful to emphasize that the cost-volume generation is per-pair (perhaps change the title of Sec 3.2) and that these volumes are averaged for all pairs.
- It might also be useful to apply the algorithm to the rectified binocular stereo case (where the warping and definition of planes by disparity are much simpler), and show comparisons to the many stereo algorithms on datasets like KITTI. At some level, the proposed algorithm can be thought of taking approaches proved to be successful for rectified binocular stereo and generalizing them (by generic warping + plane sweep) to the multi-view case. Hence, such comparisons could be illuminating. (Note: the method doesn't need to outperform the state-of-the-art there, but the results would be informative).
- I do believe the paper would significantly benefit from more discussion of DeepMVS since it's clearly the closest to this method (also solves MVS by deep networks + plane sweep). DeepMVS also learns the matching cost for cost volume generation, and the major difference seems to be that this method is learned end-to-end. It would be better to have a more detailed discussion of the differences (the current discussion at the end of Sec 2 is a little short on details)---architectures, super-vision at the end of the cost-volume vs end-to-end, etc.