Higher Order Conditional Random Fields in Deep Neural Networks 3
We evaluate our higher order potentials on the PASCAL VOC 2012 semantic segmentation benchmark as well as the PASCAL Context dataset, to show signiﬁcant
improvements over our baseline and achieve state-of-the art results.
2 Related Work
Before deep learning became prominent, semantic segmentation was performed with
dense hand-crafted features which were fed into a per-pixel or region classiﬁer [14]. The
individual predictions made by these classiﬁers were often noisy as they lacked global
context, and were thus post-processed with a CRF, making use of prior knowledge such
as the fact that nearby pixels, as well as pixels of similar appearance, are likely to share
the same class label [14, 15].
The CRF model of [14] initially contained only unary and pairwise terms in an
8-neighbourhood, which [16] showed can result in shrinkage bias. Numerous improvements to this model were subsequently proposed including: densely connected pairwise
potentials facilitating interactions between all pairs of image pixels [17], formulating
higher order potentials deﬁned over cliques larger than two nodes [5, 16] in order to
capture more context, modelling co-occurrence of object classes [18 –20], and utilising
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr