more recent FCN-based methods that have demonstrated
even better semantic segmentation results [8, 34, 16, 21].
Although these more advanced architectures could be integrated into our framework to improve our unary potentials, in this work we focus on two aspects orthogonal to
this prior work: 1) demonstrating that our boundary-based
afﬁnity function is better suited for semantic segmentation
than the common color-based afﬁnities and 2) showing that
our proposed global energy achieves better qualitative and
quantitative semantic segmentation results in comparison to
prior globalization models.
3.2. Boundary Pairwise Potentials
In this section, we describe our approach for building
pairwise pixel afﬁnities using semantic boundaries. The basic idea behind our boundary detection approach is to express semantic boundaries as a function of convolutional
feature maps inside the FCN. Due to the close relationship
between the tasks of semantic segmentation and boundary
detection, we hypothesize that convolutional feature maps
from the semantic segmentation FCN can be employed as
features for boundary detection.
3.2.1 Learning to Predict Semantic Boundaries.
We propose to express semantic boundaries as a linear combination of interpolated FCN feature maps with a non-linear
function applied on top of this sum. We note that interpolation of feature maps has been successfully used in prior
The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.
Gedas Bertasius Jianbo Shi Lorenzo Torresani