We propose DeepV2D, an end-to-end deep learning architecture for predicting depth from video. DeepV2D combines the representation ability of neural networks with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. DeepV2D interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth.
This paper pushes forward the research of deep learning based video 3D reconstruction by decomposing the problem in two-stages:
1. Depth estimation from multi-view stereo
2. Camera pose estimation from optical flow estimation and PnP SE3 pose,
which turns out to achieve state-of-the-art performance on public datasets. 
In general, the iterative procedure of this paper is similar to DeepTAM (Zhou et al., 2018), the major difference is that the camera pose is estimated by PnP with estimated Flow but not directly predicted from sub-network structures, which contributes to the higher tracking accuracy and further improves the accuracy of multi-view stereo depth estimation. 
A multi-view camera pose estimation (the global pose optimization in the paper) is also proposed to utilize the relative relation between all video frame pairs, which is difficult for fully CNN based pipeline and maximizes the use of the explicit PnP optimization. 
After that, a residual flow field is predicted using CNNs and camera poses is further refined by minimizing the geometric re-projection error.
So this paper focuses more on the camera pose estimation than the depth, which is a good start point to achieve better multi-view capabilities in a CNN framework.
However, I still have several concerns for this paper:
1. The full system is highly engineered and complicated. For example, the feature map is extracted from two hour-glass networks, which seems over-complicated for feature extraction, and the multi-view stereo network using four 3D hour-glass networks, which consumes a large amount of memories. So I would like to see the authors demonstrate the performance from in simpler settings, e.g. the feature map can be a single encoder-decoder and the multi-view stereo is done by correlation and 2D convolution. So I would like to see the inference time, peak memory consumption and the model size. Ablation studies will also help the readers to understand whether performance gain is from the pose estimation or the network capacities, which is unclear in the current paper (Appendix.D). 
2. The comparison with state-of-the-art conventional system is missing. For example, is the camera pose estimation better than initializing the DSO (Engel et al., 2018) with a monocular depth estimation? In real applications, if the performance gain is insignificant, the conventional method will still be a better choice because the CNN based methods are computationally expensive on platforms without powerful GPUs. I will not downgrade the rating if the performance gain is insignificant, but it is necessary to see the comparison.
3. Even the difference is ignorable for performance, I hope the author could use adjoint when deriving the derivatives in Eq.(11) of Appendix A.1. The author can refer to this tutorial http://ethaneade.com/lie.pdf or the text book https://www.eecis.udel.edu/~cer/arv/readings/old_mkss.pdf.
PS: For the current version with 10 pages, I lean to a borderline score, but I select 6 because 5 is not an option. I will keep or raise the current score if my concerns are addressed during rebuttal.