Practical Deep Stereo (PDS): Toward applications-friendly deep stereo matching
End-to-end deep-learning networks recently demonstrated extremely good performance for stereo matching. However, existing networks are difficult to use for practical applications since (1) they are memory-hungry and unable to process even modest-size images, (2) they have to be trained for a given disparity range. The Practical Deep Stereo (PDS) network that we propose addresses both issues: First, its architecture relies on novel bottleneck modules that drastically reduce the memory footprint in inference, and additional design choices allow to handle greater image size during training. This results in a model that leverages large image context to resolve matching ambiguities. Second, a novel sub-pixel crossentropy loss combined with a MAP estimator make this network less sensitive to ambiguous matches, and applicable to any disparity range without re-training. We compare PDS to state-of-the-art methods published over the recent months, and demonstrate its superior performance on FlyingThings3D and KITTI sets.
: Number of parameters, inference memory footprint, 3-pixels-error (3PE) and mean-absoluteerror on FlyingThings3D (960 × 540 with 192 disparities). DispNetCorr1D [20], CRL [24], iResNeti2 [18] and LRCR [12] predict disparities as classes and are consequently over-parameterized. GC [13] omits an explicit correlation step, which results in a large memory usage during inference. Our PDS has a small number of parameters and memory footprint, the smallest 3PE and smallest or second smallest MAE, depending on evaluation protocol, and it is the only method able to handle different disparity ranges without re-training. Note, that for our method we report two results. The result outside of brackets is obtained using protocol of PSM [2] method, according to which the errors are calculated only for ground truth pixel with disparity < 192. The result in the brackets is calculated according to protocol of CRL [24], DispNetCorr1D [20] and iResNet-i2 [18] methods, according to which the error is calculated only for images where less than 25% of pixels have disparity > 300, as explained in [24]. Inference memory footprints are our theoretical estimates based on network structures and do not include memory required for storing networks' parameters (real memory footprint will depend on implementation). Error rates and numbers of parameters are taken from the respective publications.
Summary -------  This paper presents a method for computing a disparity map from a rectified pair of images. The authors propose an end-to-end learning approach consisting of three modules: the embedding module, the matching module, and the regularization module. The embedding module extracts feature vectors from the image par. The feature vectors from the left image and right image are horizontally offset and concatenated (this is repeated for all disparities under consideration) and the matching module is applied to compress the concatenated feature vectors. The resulting volume is processed by the regularization module which returns a distribution over disparities for each spatial location. The network is trained on full-resolution images. The target for training is a discretiezed Laplace distribution centered around the ground truth disparity and the cross-entropy loss is used to train the network.  At inference the disparity is computed using sub-pixel MAP approximation, which is defined as the expected value of disparity taking into consideration only disparity values around the disparity with maximum posterior probability.  The parts of the architecture that are novel are the matching module, sub-pixel MAP approximation, and using a discretized Laplace distribution as target.  Strengths ---------  - The authors submitted the generated disparity maps to the KITTI online   evaluation server. The method performs well and is bested by only a couple of   recently published papers. The KITTI leaderboard is very competitive and   placing 3rd and 4th is respectable. The results on KITTI are the main reason    why I leaned towards accepting the paper.  - The code will be open-sourced after publication, which makes it easy to   reproduce the results and build upon the ideas presented in this work.  - Ablation studies on the sub-pixel MAP estimator are performed.  Weaknesses ----------  - The contributions of this work (the matching module, sub-pixel MAP   approximation, and using a discretized Laplace distribution during training)   are solid, but not ground-breaking.  Comments --------  Line 62 and line 153: The authors say that using cross-entropy is more "natural" than using L1 loss. Can you explain what you mean by natural?  Line 70: "training with a vanilla L1 penalty". I'm not sure what the word "vanilla" is referring to. Maybe the authors should consider omitting it?  In equation after line 91 the authors apply the softmax function to a three dimensional tensor, whereas the standard definition softmax is on vectors. My guess is that the softmax function is applied on each spatial location. Perhaps the text could mention this or the equation should be written more precisely.  Line 181: "by processing the test set using the ground truth for benchmarking". I'm slightly confused by the meaning of this sentence. My interpretation was that that "they processed the test set, which is used for benchmarking".  Line 182 "without mentioning it" and line 184 "although this is not commendable". I would consider omitting these two sentences, because they  don't really convey any meaning and come across as slightly negative.  Figure 4 and line 210: "Note that SoftArgmin estimate, though completely wrong, is closer to the ground truth than sub-pixel MAP estimate". Maybe consider omitting the work "completely" or re-editing this sentence. After all, If an estimate is completely wrong it can't be better than another estimate.  Spelling --------  Line 46: "It composed of" -> "It is composed of" Line 47: "and not treat" -> "and does? not treat" Line 51: "their disparity range is still is non-adjustable" -> "their disparity range is still non-adjustable" Line 61: "it allows to train network to" -> "it allows to train the network to" Line 68: "to train on a full-size images" -> "to train on full-size images" Line 80: "Than, in 3.4 we" -> "Then, in 3.4 we" Line 205: "we firstly show" -> "we first? show" Line 206: "we train the our PDS network" -> "we train our PDS network" Line 210: "SoftArgmin estimate are" -> "SoftArgmin estimates? are" Figure 2: "Comparison the proposed" -> "Comparison of? the proposed" Figure 4: "The first column shows image" -> "The first column shows the? input? image"