are examples where superpixel potentials help to remove spurious noise in the output
but detection potentials do not affect the result. The ﬁnal result still improves when
all potentials are combined. (Row 3) Detection potentials greatly improve the result
by recognising the train correctly (the pixelwise unaries are largest for “bus”). And
superpixels, when combined with detections, slightly improve the output. (Row 4) An
example where both superpixel and detection potentials improve the ﬁnal output. (Row 5)
A case where the superpixel worsens the result as, although the output is more consistent
among superpixel regions, some pixels have had their correct labels removed. However,
the correct detection improves the result, and the output of combining superpixel and
detection potentials is actually better than either potential in isolation. (Row 6) Here, the
detection (although correct) worsens the output due to its imprecise foreground mask.
Superpixel potentials also exacerbate the result, since the legs of the chair and the chair’s
shadow are confused to be part of the same superpixel region. However, when the two
potentials are combined, the result is slightly better than with only detection potentials.
24 Arnab et al.
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr