ﬁrst, “unary” part of our network is formed by the FCN-8s architecture [3]. It is initialised
from the Imagenet-trained VGG-16 network [2], and then ﬁne-tuned with data from
the VOC 2012 training set [4], extra VOC annotations of [43] and the MS COCO [44]
dataset.
Higher Order Conditional Random Fields in Deep Neural Networks 11
Table 1: Comparison of each higher order
potential with baseline on VOC 2012 reduced validation set
MethodReduced
val set(%)
Baseline (unary + pairwise) [10] 72.9
Superpixels only 74.0
Detections only 74.9
Detections and Superpixels 75.8Table 2: Mean IoU accuracy on VOC 2012
test set. All methods are trained with MS
COCO [44] data
Method Test set(%)
Ours 77.9
DPN [9] 77.5
Centrale Super Boundaries [45] 75.7
Dilated Convolutions [46] 75.3
BoxSup [35] 75.2
DeepLab Attention [47] 75.1
CRF-RNN (baseline) [10] 74.7
DeepLab WSSL [48] 73.9
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr