and pairwise potentials. Minimization of the global energy yields the semantic segmentation. BNFs build both
unary and pairwise potentials from the input RGB image
and then combine them in a global manner. More precisely,
the coarse segmentations predicted by a semantic FCN are
used to deﬁne the unary potentials of our BNF. Next, we
show that the convolutional feature maps of the FCN can
be used to accurately predict semantic boundaries. These
boundaries are then used to build pairwise pixel afﬁnities,
which are used as pairwise potentials by the BNF. Finally,we introduce a global energy function, which minimizes the
energy corresponding to the unary and pairwise terms and
improves the initial FCN segmentation. The detailed illustration of our architecture is presented in Figure 2. We now
explain each of these steps in more detail.
3.1. FCN Unary Potentials
To predict semantic unary potentials we employ the
DeepLab model [7], which is a fully convolutional adaptation of the VGG network [30]. The FCN consists of 16convolutional layers and 3fully convolutional layers. There are
more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].
The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.
Gedas Bertasius Jianbo Shi Lorenzo Torresani