learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld
inference with higher order potentials, they did not consider object detection potentials
at all, nor were the parameters learned.
3 Conditional Random Fields
We now review conditional random ﬁelds used in semantic segmentation and introduce
the notation used in the paper. Take an image IwithNpixels, indexed 1,2,...,N . In
semantic segmentation, we attempt to assign every pixel a label from a predeﬁned set of
labels L={l1,l2,...,l L}. Deﬁne a set of random variables X1,X2,...,X N, one for
each pixel, where each Xi∈ L. LetX= [X1X2... X N]T. Any particular assignment
xtoXis thus a solution to the semantic segmentation problem.
We use notations {V}, andV(i)to represent the set of elements of a vector V, and
theithelement of V, respectively. Given a graph Gwhere the vertices are from {X}
and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr