classify each pixel as belonging to a certain class. While
these approaches partially address the incorrect segmentation problem, they perform predictions independently on
each pixel. This leads to extremely local predictions, where
the relationships between pixels are not exploited in any
way, and thus the resulting segmentations may be spatially
disjoint.
The third and ﬁnal group of semantic segmentation
methods can be viewed as front-to-end schemes where segmentation maps are predicted directly from raw pixels without any intermediate steps. One of the earliest examples of
such methods is the FCN introduced in [22]. This approach
gave rise to a number of subsequent related approaches
which have improved various aspects of the original semantic segmentation [7, 34, 8, 16, 21]. There have also been attempts at integrating the CRF mechanism into the network
architecture [7, 34]. Finally, it has been shown that semantic
segmentation can also be improved using additional training
data in the form of bounding boxes [8].
Our BNF offers several contributions over prior work. To
the best of our knowledge, we are the ﬁrst to present a model
that exploits the relationship between boundary detection
and semantic segmentation within a FCN framework . We
introduce pairwise pixel afﬁnities computed from semantic boundaries inside an FCN, and use these boundaries to
The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.
Gedas Bertasius Jianbo Shi Lorenzo Torresani