capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to
have consistent labelling. These two new potentials can co-exist with the usual unary
and pairwise potentials in a CRF.
Importantly, we showed that efﬁcient mean ﬁeld inference is still possible in the
presence of the new higher order potentials and derived the explicit forms of the mean
ﬁeld updates and their differentials. This enabled us to implement the new CRF model
as a stack of CNN layers and to train it end-to-end in a uniﬁed deep network with a
pixelwise CNN classiﬁer. We experimentally showed that the addition of higher order
potentials results in a signiﬁcant increase in semantic segmentation accuracy allowing us
to reach state-of-the-art performance.
This work was supported by ERC grant ERC-2012-AdG 321162-HELIOS, EPSRC grant
Seebibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1 and the Clarendon Fund.
Higher Order Conditional Random Fields in Deep Neural Networks 15
References
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr