boundary points in a pre-processing stage, which was instead required in prior boundary detection work [3, 4].
Our boundary prediction pipeline can be described as
follows. First we use use SBD segmentations [13] to optimize our FCN for semantic segmentation task. We then
treat FCN convolutional maps as features for the boundary detection task and use the boundary annotations from
BSDS 500 dataset [23] to learn the weights for each feature
map. BSDS 500 dataset contains 200training, 100validation, 200testing images, and ground truth annotations by 5
human labelers for each of these images.
To learn the weights corresponding to each convolutional
feature map we ﬁrst sample 80Kpoints from the dataset.
We deﬁne the target labels for each point as the fraction of
human annotators agreeing on that point being a boundary.
To ﬁx the issue of label imbalance (there are many more
non-boundaries than boundaries), we divide the label space
into four quartiles, and select an equal number of samples
for each quartile to balance the training dataset. Given these
sampled points, we then deﬁne our features as the values in
the interpolated convolutional feature maps corresponding
to these points. To predict semantic boundaries we weigh
each convolutional feature map by its weight, sum them up
The state-of-the-art in semantic segmentation is currently represented by
fully convolutional networks (FCNs). However, FCNs use large receptive fields
and many pooling layers, both of which cause blurring and low spatial
resolution in the deep layers. As a result FCNs tend to produce segmentations
that are poorly localized around object boundaries. Prior work has attempted to
address this issue in post-processing steps, for example using a color-based
CRF on top of the FCN predictions. However, these approaches require additional
parameters and low-level features that are difficult to tune and integrate into
the original network architecture. Additionally, most CRFs use color-based
pixel affinities, which are not well suited for semantic segmentation and lead
to spatially disjoint predictions.
  To overcome these problems, we introduce a Boundary Neural Field (BNF), which
is a global energy model integrating FCN predictions with boundary cues. The
boundary information is used to enhance semantic segment coherence and to
improve object localization. Specifically, we first show that the convolutional
filters of semantic FCNs provide good features for boundary detection. We then
employ the predicted boundaries to define pairwise potentials in our energy.
Finally, we show that our energy decomposes semantic segmentation into multiple
binary problems, which can be relaxed for efficient global optimization. We
report extensive experiments demonstrating that minimization of our global
boundary-based energy yields results superior to prior globalization methods,
both quantitatively as well as qualitatively.
Gedas Bertasius Jianbo Shi Lorenzo Torresani