Learning a Multi-View Stereo Machine
We present a learnt system for multi-view stereopsis. In contrast to recent learning based methods for 3D reconstruction, we leverage the underlying 3D geometry of the problem through feature projection and unprojection along viewing rays. By formulating these operations in a differentiable manner, we are able to learn the system end-to-end for the task of metric 3D reconstruction. End-to-end learning allows us to jointly reason about shape priors while conforming to geometric constraints, enabling reconstruction from much fewer images (even a single image) than required by classical approaches as well as completion of unseen surfaces. We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches and recent learning based methods.
This submission proposes to learn 3D object reconstruction from a multi-view stereo setting end-to-end. Its main contribution is consistent feature projection and unprojection along viewing rays given camera poses and multiple images. Instead of mainly learning object shapes based on semantic image interpretation, this approach builds projective geometry into the approach to benefit from geometric cues, too. The paper proposes a way for differentiable unprojection that projects 2D image features to 3D space using camera projection equations and bilinear sampling. This is a simple yet effective way to implicitly build epipolar constraints into the system, because features from multiple views of the same world point unproject to the same voxel. While such combination of multiple views is of course not new, it is new as part of an end-to-end learnable approach.
I did not completely understand how recurrent grid fusion works, please explain in more detail. A dedicated figure might ease understanding, too. I would also appreciate a short note on GPU RAM requirements and run-time.   
The paper is very well written and illustrations make it enjoyable to read. The approach is experimentally evaluated by comparison to traditional multi-view stereo matching algorithms (e.g., plane sweeping) and it is compared to the recent 3D-R2N2 approach, that also uses deep learning for 3D object reconstruction. 
While all experiments are convincing and support claims of the paper, it would be good to compare to more deep learning-based 3D reconstruction approaches (several are cited in the paper, e.g. [27] + additional work like "DeMoN: Depth and Motion Network for Learning Monocular Stereo" from Thomas Brox's lab). Clear explanation of theoretical differences to [27] would further improve the paper. At present, it remains unclear whether this submission is just a multi-view extension of [27] without reading [27] itself. Better motivating the own contribution here seems necessary. 
I believe this is a great paper and deserves to be published at NIPS, but also must admit that I still have some questions.