row shows an example of how the detection of the person has now resulted in the sofa
being misclassiﬁed (although our system is able to reject the other false detection). Our
superpixel potentials have a tendency to remove spurious noise by enforcing consistency
within regions. However, as shown in the second row, sometimes the “noise” being
removed is actually the correct label. In the other cases, we are limited by our pixelwise
classiﬁcation unaries which are poor. Our superpixel and detection potentials are not
always able to compensate for this. Note that all images are from the reduced validation
set of VOC 2012 and have not been trained on at all. The input images have the detection
bounding boxes overlaid on them. Note that the method of [10] does not make use of
this information. Best viewed in color.
Higher Order Conditional Random Fields in Deep Neural Networks 23
Input image Pairwise onlySuperpixels
onlyDetections
onlyDetections
and
SuperpixelsGround truth
Fig. 8: Comparison of pairwise potentials, superpixel and pairwise potentials, detection and pairwise potentials, and a combination of all three (Row 1 and 2) These
are examples where superpixel potentials help to remove spurious noise in the output
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr