thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.
Object detectors have also been used by [21, 25], who also modelled variables that
describe the degree to which an object hypothesis is accepted.
We formulate the detection potential in a different manner to [6, 21, 25] so that it is
amenable to mean ﬁeld inference. Mean ﬁeld permits inference with dense pairwise connections, which results in substantial accuracy improvements [10,12, 17]. Furthermore,
mean ﬁeld updates related to our potentials are differentiable and its parameters can thus
be learned in our end-to-end trainable architecture.
We also note that while the semantic segmentation problem has mostly been formulated in terms of pixels [3,10,14], some have expressed it in terms of superpixels [26 –28].
Superpixels can capture more context than a single pixel and computational costs can
also be reduced if one considers pairwise interactions between superpixels rather than individual pixels [21]. However, such superpixel representations assume that the segments
4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr