image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference
is impracticable, and one has to resort to an approximate inference method such as mean
ﬁeld inference [17]. Mean ﬁeld inference is particularly appealing in a deep learning
setting since it is possible to formulate it as a Recurrent Neural Network [10].
Higher Order Conditional Random Fields in Deep Neural Networks 5
4 CRF with Higher Order Potentials
Many CRF models that have been incorporated into deep learning frameworks [10, 12]
have so far used only unary and pairwise potentials. However, potentials deﬁned on
higher order cliques have been shown to be useful in previous works such as [7, 16]. The
key contribution of this paper is to show that a number of explicit higher order potentials
can be added to CRFs to improve image segmentation, while staying compatible with
deep learning. We formulate these higher order potentials in a manner that mean ﬁeld
inference can still be used to solve the CRF. Advantages of mean ﬁeld inference are
twofold: First, it enables efﬁcient inference when using densely-connected pairwise
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr