foreground mask are also largely ignored.
4.2 Superpixel Based Potentials
The next type of higher order potential we use is based on the idea that superpixels
obtained from oversegmentation [39, 40] quite often contain pixels from the same visual
object. It is therefore natural to encourage pixels inside a superpixel to have the same
semantic label. Once again, this should not be a hard constraint in order to keep the
8 Arnab et al.
(a) (b) (c) (d)
Fig. 3: Effects of imperfect foreground segmentation (a,b) Detected objects, as well
as the foreground masks obtained from GrabCut. (c,d) Output using detection potentials.
Incorrect parts of the foreground segmentation of the main aeroplane, and entire TV
detection have been ignored by CRF inference as they did not agree with the other energy
terms. The person is a failure case though as the detection has caused part of the sofa to
be erroneously labelled.
(a) (b) (c)
Fig. 4: Segmentation enhancement from superpixel based potentials (a) The output
of our system without any superpixel potentials. (b) Superpixels obtained from the image
using the method of [39]. Only one “layer” of superpixels is shown. In practice, we
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr