parameters that are difﬁcult to tune and integrate into the
original network architecture. Additionally, most methods
based on CRFs or MRFs use low-level pixel afﬁnity functions, such as those based on color. These low-level afﬁnities often fail to capture semantic relationships between objects and lead to poor segmentation results (see last column
in Fig. 1).
We propose to address these shortcomings by means of
a Boundary Neural Field (BNF), an architecture that employs a single semantic segmentation FCN to predict semantic boundaries and then use them to produce semantic
segmentation maps via a global optimization. We demonstrate that even though the semantic segmentation FCN has
not been optimized to detect boundaries, it provides good
1arXiv:1511.02674v2  [cs.CV]  24 May 2016
...
Convolutional Feature Maps RGB Input Image
Segmentation Mask in       Softmax Layer
   Minimize Global Energy
Boundaries
|{z}...
wn()++()w1
Xi✓i(zi)+Xij✓ij(zi,zj)
Linear Combination of Feature MapsFigure 2: The architecture of our system (best viewed in
color). We employ a semantic segmentation FCN [7] for
two purposes: 1) to obtain semantic segmentation unaries
The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.
Gedas Bertasius Jianbo Shi Lorenzo Torresani