i(.)andψP
ij(.,.)are the usual unary and densely-connected
pairwise energies [17] and the last two terms are the newly introduced higher order
energies. Energies from the object detection take the form ψDet
d(xd), where vector xd
is formed by elements of xthat correspond to the foreground pixels of the dthobject
detection. Superpixel label consistency based energies take the form ψSP
s(xs), where xs
is formed by elements of xthat correspond to the pixels belonging to the sthsuperpixel.
4.1 Object Detection Based Potentials
Semantic segmentation errors can be classiﬁed into two broad categories [35]: recognition
and boundary errors. Boundary errors occur when semantic labels are incorrect at the
edges of objects, and it has been shown that densely connected CRFs with appearanceconsistency terms are effective at combating this problem [17]. On the other hand,
recognition errors occur when object categories are recognised incorrectly or not at
all. A CRF with only unary and pairwise potentials cannot effectively correct these
errors since they are caused by poor unary classiﬁcation. However, we propose that a
state-of-the-art object detector [36,37] capable of recognising and localising objects, can
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr