Recursive Context Propagation Network for Semantic Scene Labeling
We propose a deep feed-forward neural network architecture for pixel-wise semantic scene labeling. It uses a novel recursive neural network architecture for context propagation, referred to as rCPN. It first maps the local visual features into a semantic space followed by a bottom-up aggregation of local information into a global representation of the entire image. Then a top-down propagation of the aggregated information takes place that enhances the contextual information of each local feature. Therefore, the information from every location in the image is propagated to every other location. Experimental results on Stanford background and SIFT Flow datasets show that the proposed method outperforms previous approaches. It is also orders of magnitude faster than previous methods and takes only 0.07 seconds on a GPU for pixel-wise labeling of a 256 × 256 image starting from raw RGB pixel values, given the super-pixel mask that takes an additional 0.3 seconds using an off-the-shelf implementation.
In this paper, the authors propose a model called 'rCPN' composed of a sequence of neural networks for semantic scene labeling (label every pixel in a scene by the class it belongs to). They use the network proposed by Farabet et al. (multiscale CNN, with some small differences such as using ReLU non-linearity and Dropout) to extract features that describe local characteristic of superpixels. These local featuresare then passed through a network to map them into a semantic space where local information can be propagated. This local information is then recursively aggregated using a parse tree hierarchy. Once the scene is completely aggregated, a third network is used to disseminate the holistic description back into the initial superpixel segmentation. Finally, a fourth network is used to classify the superpixels into different classes.The training is done into two separate stages: first the mutliscale CNN is trained from raw pixel. Then the image is segmented into a set of superpixels, and binary parse trees are made. Finally, rCPN (consisting of the four networks) are trained to predict the label of each superpixel.The paper is well written in general and the ideas are easy to get. The Related Works section could be inserted between Introduction and Learning sections instead. It would be interesting to make a better comparison with other models also using neural networks in the Related Works section (tradeoff between improved results/complexity of model). For the Experiments sections, the Stanford Background dataset results could be introduced before the SIFT Flow dataset ones, as it is a simpler dataset somehow. In line 259, when comparing the training time with Farabet et al you should mention you do GPU training while Farabet et al. explicitly mention no use of GPU. Also, in Table 1, you add the inference time for Pinheiro et al which I couldn't find in the cited paper, maybe you should just leave NA as in the others.The authors achieve good results with the expense of a complex, non-straightforward model (many different steps during the training).For the experiments, maybe it would be interesting to mention the other results from Farabet (pure multi-CNN and multi-CNN/superpixel) and Pinheiro (considering smaller output resolution) with faster models, which achieve relatively good results with faster inference time (and a more straightforward models trained from raw data in one step). Also, how many parameters does the model has, compared to other approaches?