the superpixel. Costs wLow(l)andwHigh are learnable during the end-to-end training of
the network.
Secondly, to make this potential stronger, we average initial unary potentials from the
classiﬁer (the CNN in our case), across all pixels in the superpixel and use the average
Higher Order Conditional Random Fields in Deep Neural Networks 9
as an additional unary potential for those pixels. During experiments, we observed that
superpixel based higher order energy helps in getting rid of small spurious regions of
wrong labels in the segmentation output, as shown in Fig. 4.
5 Mean Field Updates and Their Differentials
This section discusses the mean ﬁeld updates for the higher order potentials previously
introduced. These update operations are differentiable with respect to the Qi(Xi)distribution inputs at each iteration, as well as the parameters of our higher order potentials.
This allows us to train our CRF end-to-end as another layer of a neural network.
Take a CRF with random variables V1,V2,...,V Nand a set of cliques C, which
includes unary, pairwise and higher order cliques. Mean ﬁeld inference approximates
the joint distribution Pr(V=v)with the product of marginals/producttext
iQ(Vi=vi). We
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr