and the edges deﬁne connections among these variables, the pair (I,X)is modelled as a
CRF characterised by Pr(X=x|I) = (1/Z(I)) exp( −E(x|I)), whereE(x|I)is the
energy of the assignment xandZ(I)is the normalisation factor known as the partition
function. We drop the conditioning on Ihereafter to keep the notation uncluttered. The
energyE(x)of an assignment is deﬁned using the set of cliques Cin the graph G. More
speciﬁcally, E(x) =/summationtext
c∈Cψc(xc),where xcis a vector formed by selecting elements
ofxthat correspond to random variables belonging to the clique c, andψc(.)is the
cost function for the clique c. The function, ψc(.), usually uses prior knowledge about a
good segmentation, as well as information from the image, the observation the CRF is
conditioned on.
Minimising the energy yields the maximum a posteriori (MAP) labelling of the
image i.e.the most probable label assignment given the observation (image). When
dense pairwise potentials are used in the CRF to obtain higher accuracy, exact inference
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr