Convolutional Neural Networks with Intra-layer Recurrent Connections for Scene Labeling
Scene labeling is a challenging computer vision task. It requires the use of both local discriminative features and global context information. We adopt a deep recurrent convolutional neural network (RCNN) for this task, which is originally proposed for object recognition. Different from traditional convolutional neural networks (CNN), this model has intra-layer recurrent connections in the convolutional layers. Therefore each convolutional layer becomes a two-dimensional recurrent neural network. The units receive constant feed-forward inputs from the previous layer and recurrent inputs from their neighborhoods. While recurrent iterations proceed, the region of context captured by each unit expands. In this way, feature extraction and context modulation are seamlessly integrated, which is different from typical methods that entail separate modules for the two steps. To further utilize the context, a multi-scale RCNN is proposed. Over two benchmark datasets, Standford Background and Sift Flow, the model outperforms many state-of-the-art models in accuracy and efficiency.
This paper presents a recurrent convolutional neural network for semantic image segmentation to encode and take advantage of contextual relationships. The method is basically a combination of [13] where a very similar RCNN is used for object recognition, and [4] from which the multi-scale pipeline is inspired. Thus, technically, the paper is not very novel (Sec. 3.1. is much the same as in previous works - maybe that should be stated more clearly; the combination can be seen as a novelty of course). However, the paper is well executed, very easy and clear to read, largely well written and provides a seemingly fair evaluation to state-of-the-art. Some recent works or interesting evaluations could be added, see below. Results are shown on the Sift Flow and the Stanford Background dataset where the proposed technique outperforms state-of-the-art using a limited number of training data for all approaches (limited by these datasets). Also, parameter ablation studies are conducted to some extend. The method is very efficient.Detailed comments:- The related work section could be written more sharply. It is not always crystal clear what the differences are with respect to the closest related works.- Results on PASCAL VOC could be added as most recent segmentation works evaluate on those datasets.- l.276 is very vague and should be made more clear.- Table 1: Why are only \gamma \in {0,1} considered? The paper could provide experiments/plots with varying gammas.- page 7: The ordering of text/plots/tables should be changed to have less interleaving text/plots/tables.- The model mentioned in l.387 should be added to the table, maybe with a footnote that it uses different training data or a line break.- Some further qualitative analysis would be nice if it fits.- This recent work seems to be missing (the first one provides even stronger results on Stanford Background): Feedforward semantic segmentation with zoom-out features Mohammadreza Mostajabi, Payman Yadollahpour and Gregory Shakhnarovich Toyota Technological Institute at Chicago@inproceedings{crfasrnn_arXiv2015,author = {Shuai Zheng and Sadeep Jayasumana and Bernardino Romera-Paredes and Vibhav Vineet and Zhizhong Su and Dalong Du and Chang Huang and Philip Torr},title = {Conditional Random Fields as Recurrent Neural Networks},booktitle = {arXiv:1502.03240}, year = {2015}}