the matrix multiplication efﬁcient and exact, but it is also
easy to integrate into the traditional FCN framework for
both training and prediction schemes. Additionally, due to
the use of standard convolution and matrix multiplication
operations, our RWN can be trivially trained via standard
back-propagation in an end-to-end fashion.
4. Convolutional Random Walk Networks
In this work, our goal is to integrate a random walk process into the FCN architecture to encourage coherent semantic segmentation among pixels that are similar to each
other. Such a process introduces an explicit grouping mechanism, which should be beneﬁcial in addressing the issues
of (1) poor localization around the boundaries, and (2) spatially fragmented segmentations.
A schematic illustration of our proposed RWN architecture is presented in Fig. 2. Our RWN is a network composed of two branches: (1) one branch that predicts semantic segmentation potentials, and (2) another branch devoted
to predicting pixel-level afﬁnities. These two branches are
merged via a novel random walk layer that encourages spatially coherent semantic segmentation. The entire RWN can
be jointly optimized end-to-end. We now describe each of
the components of the RWN architecture in more detail.
4.1. Semantic Segmentation Branch
For the semantic segmentation branch, we present results
Most current semantic segmentation methods rely on fully convolutional
networks (FCNs). However, their use of large receptive fields and many pooling
layers cause low spatial resolution inside the deep layers. This leads to
predictions with poor localization around the boundaries. Prior work has
attempted to address this issue by post-processing predictions with CRFs or
MRFs. But such models often fail to capture semantic relationships between
objects, which causes spatially disjoint predictions. To overcome these
problems, recent methods integrated CRFs or MRFs into an FCN framework. The
downside of these new models is that they have much higher complexity than
traditional FCNs, which renders training and testing more challenging.
  In this work we introduce a simple, yet effective Convolutional Random Walk
Network (RWN) that addresses the issues of poor boundary localization and
spatially fragmented predictions with very little increase in model complexity.
Our proposed RWN jointly optimizes the objectives of pixelwise affinity and
semantic segmentation. It combines these two objectives via a novel random walk
layer that enforces consistent spatial grouping in the deep layers of the
network. Our RWN is implemented using standard convolution and matrix
multiplication. This allows an easy integration into existing FCN frameworks
and it enables end-to-end training of the whole network via standard
back-propagation. Our implementation of RWN requires just $131$ additional
parameters compared to the traditional FCNs, and yet it consistently produces
an improvement over the FCNs on semantic segmentation and scene labeling.
Gedas Bertasius Lorenzo Torresani Stella X. Yu Jianbo Shi