are obtained from the original semantic segmentation anInput
 DeepLab_v2
 RWN_v2
Figure 3: A ﬁgure illustrating the segmentation results of
our RWN and the DeepLab-v2 network. Note that RWN
produced segmentations are spatially smoother and produce
less false positive predictions than the DeepLab-v2 system.
notations. Speciﬁcally, we set the ground truth afﬁnity between two pixels to 1if the pixels share the same semantic
label and have distance less than Rfrom each other. Note
thatF, which is used as an input to the afﬁnity branch, is a
sparse matrix, as only a small fraction of all the entries in F
are populated with non-zero values. The rest of the entries
are ignored during the computation.
Also note that we only use features from RGB, conv1_1
andconv1_2layers, because they are not affected by pooling, and thus, preserve the original spatial resolution. We
also experimented with using features from deeper FCN
layers such as fc6, and fc7. However, we observed that
features from deeper layers are highly correlated to the
predicted semantic segmentation unary potentials, which
causes redundancy and little improvement in the segmentation performance. We also experimented with using more
Most current semantic segmentation methods rely on fully convolutional
networks (FCNs). However, their use of large receptive fields and many pooling
layers cause low spatial resolution inside the deep layers. This leads to
predictions with poor localization around the boundaries. Prior work has
attempted to address this issue by post-processing predictions with CRFs or
MRFs. But such models often fail to capture semantic relationships between
objects, which causes spatially disjoint predictions. To overcome these
problems, recent methods integrated CRFs or MRFs into an FCN framework. The
downside of these new models is that they have much higher complexity than
traditional FCNs, which renders training and testing more challenging.
  In this work we introduce a simple, yet effective Convolutional Random Walk
Network (RWN) that addresses the issues of poor boundary localization and
spatially fragmented predictions with very little increase in model complexity.
Our proposed RWN jointly optimizes the objectives of pixelwise affinity and
semantic segmentation. It combines these two objectives via a novel random walk
layer that enforces consistent spatial grouping in the deep layers of the
network. Our RWN is implemented using standard convolution and matrix
multiplication. This allows an easy integration into existing FCN frameworks
and it enables end-to-end training of the whole network via standard
back-propagation. Our implementation of RWN requires just $131$ additional
parameters compared to the traditional FCNs, and yet it consistently produces
an improvement over the FCNs on semantic segmentation and scene labeling.
Gedas Bertasius Lorenzo Torresani Stella X. Yu Jianbo Shi