IoU” is evaluated outside this region.
Fig. 5 shows our results as the trimap width is varied. Adding the detection potentials
improves the Interior IoU over our baseline (only pairwise potentials [10]) as the object
detector may recognise objects in the image which the pixelwise classiﬁcation stage of
our network may have missed out. However, the detection potentials also improve the
Boundary IoU for all tested trimap widths as well. Improving the recognition of pixels
in the interior of an object also helps with delineating the boundaries since the strength
of the pairwise potentials exerted by the Qdistributions at each of the correctly-detected
pixels increase.
Our superpixel priors also increase the Interior IoU with respect to the baseline.
Encouraging consistency over regions helps to get rid of spurious regions of wrong
labels (as shown in Fig. 4). Fig. 5 suggests that most of this improvement occurs in the
interior of an object. The Boundary IoU is slightly lower than the baseline, and this may
Higher Order Conditional Random Fields in Deep Neural Networks 13
a) Image (b) Ground truth (c) Boundary (d) Interior
0 5 10 15 20 25 30 35 406668707274767880
Trimap WidthInterior IoU
  
Pairwise only
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr