using the method of [39]. Only one “layer” of superpixels is shown. In practice, we
used four. (c) The output using superpixel potentials. The result has improved as we
encourage consistency over superpixel regions. This removes some of the spurious noise
that was present previously.
algorithm robust to initial superpixel segmentation errors and to violations of this key
assumption.
We use two types of energies in the CRF to encourage superpixel consistency in
semantic segmentation. Firstly, we use the Pn-Potts model type energy [41], which is
described by,
ψSP
s(Xs=xs) =

wLow(l) if allx(i)
s=l,
wHigh otherwise,(4)
wherewLow(l)<w High for alll, and{Xs} ⊂ {X}is a clique deﬁned by a superpixel.
The primary idea is that assigning different labels to pixels in the same superpixel incurs
a higher cost, whereas one obtains a lower cost if the labelling is consistent throughout
the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr