Inspired by the combination of feedforward and iterative computations in the visual cortex, and taking advantage of the ability of denoising autoencoders to estimate the score of a joint distribution, we propose a novel approach to iterative inference for capturing and exploiting the complex joint distribution of output variables conditioned on some input variables. This approach is applied to image pixel-wise segmentation, with the estimated conditional score used to perform gradient ascent towards a mode of the estimated conditional distribution. This extends previous work on score estimation by denoising autoencoders to the case of a conditional distribution, with a novel use of a corrupted feedforward predictor replacing Gaussian corruption. An advantage of this approach over more classical ways to perform iterative inference for structured outputs, like conditional random fields (CRFs), is that it is not any more necessary to define an explicit energy function linking the output variables. To keep computations tractable, such energy function parametrizations are typically fairly constrained, involving only a few neighbors of each of the output variables in each clique. We experimentally find that the proposed iterative inference from conditional score estimation by conditional denoising autoencoders performs better than comparable models based on CRFs or those not using any explicit modeling of the conditional joint distribution of outputs.
The paper proposes an image segmentation method which iteratively refines the semantic segmentation mask obtained from a deep net. To this end the authors investigate a denoising auto-encoder (DAE). Its purpose is to provide a semantic segmentation which improves upon its input in terms of the log-likelihood.
More specifically, the authors `propose to condition the autoencoder with an additional input’ (page 1). To this end they use features obtained from the deep net. Instead of training the DAE with ground truth y, the authors found usage of the deep net prediction to yield better results.
The proposed approach is evaluated on the CamVid dataset.
Summary:
——
I think the paper discusses a very interesting topic and presents an elegant approach. A few points are missing which would provide significantly more value to a reader. Specifically, an evaluation on the classical Pascal VOC dataset, details regarding the training protocol of the baseline (which are omitted right now), an assessment regarding stability of the proposed approach (not discussed right now), and a clear focus of the paper on segmentation or conditioning. See comments below for details and other points.
Comments:
——
1. When training the DAE, a combination of squared loss and categorical cross-entropy loss is used. What’s the effect of the squared error loss and would the categorical cross-entropy on its own be sufficient? This question remains open when reading the submission.
2. The proposed approach is evaluated on the CamVid dataset which is used less compared to the standard and larger Pascal VOC dataset. I conjecture that the proposed approach wouldn’t work too well on Pascal VOC. On Pascal VOC, images are distinctly different from each other whereas subsequent frames are similar in CamVid, i.e., the road is always located at the bottom center of the image. The proposed architecture is able to take advantage of this dataset bias, but would fail to do so on Pascal VOC, which has a much more intricate bias. It would be great if the authors could check this hypothesis and report quantitative results similar to Tab. 1 and Fig. 4 for Pascal VOC.
3. The authors mention a grid-search for the stepsize and the number of iterations. What values were selected in the end on the CamVid and hopefully the Pascal VOC dataset?
4. Was the dense CRF applied out of the box, or were its parameters adjusted for good performance on the CamVid validation dataset? While parameters such as the number of iterations and epsilon are tuned for the proposed approach on the CamVid validation set, the submission doesn’t specify whether a similar procedure was performed for the CRF baseline.
5. Fig. 4 seems to indicate that the proposed approach doesn’t converge. Hence an appropriate stepsize and a reasonable number of iterations need to be chosen on a validation set. Choosing those parameters guarantees that the method performs well on average, but individual results could potentially be entirely wrong, particularly if large step sizes are chosen. I suspect this effect to be more pronounced on the Pascal VOC dataset (hence my conjecture in point 2). To further investigate this property, as a reader, I’d be curious to get to know the standard deviation/variance of the accuracy in addition to the mean IoU. Again, it would be great if the authors could check this hypothesis and report those results.
6. I find the experimental section to be slightly disconnected from the initial description. Specifically, the paper `proposes to condition the autoencoder with an additional input’ (page 1). No experiments are conducted to validate this proposal. Hence the main focus of the paper (image segmentation or DAE conditioning) remains vague. If the authors choose to focus on image segmentation, a comparison to state-of-the-art should be provided on classical datasets such as Pascal VOC, if DAE conditioning is the focus, some experiments in this direction should be included in addition to the Pascal VOC results.
Minor comment:
——
- I find it surprising that the authors choose not to cite some related work on combining deep nets with structured prediction.