on the VOC benchmark increases with higher order potentials.
Beneﬁts of end-to-end training Table 4 shows how end-to-end training outperforms
piecewise training. We trained the CRF piecewise by freezing the weights of the unary
part of the network, and only learning the CRF parameters.
Our results in Table 2 used the FCN-8s [3] architecture to generate unaries. To show
that our higher order potentials improve performance regardless of the underlying CNN
used for producing unaries, we also perform an experiment using our reimplementation
Table 4: Comparison of mean IoU (%) obtained on VOC 2012 reduced validation set
from end-to-end and piecewise training
Method FCN-8s DCN
Unary only, ﬁne-tuned on COCO 68.3 68.6
Pairwise CRF trained piecewise 69.5 70.7
Pairwise CRF trained end-to-end 72.9 72.5
Higher Order CRF trained piecewise 73.6 73.5
Higher Order CRF trained end-to-end 75.8 75.0
Test set performance of best model 77.9 76.9
14 Arnab et al.
of the “front-end” module proposed in the Dilated Convolution Network (DCN) of [46]
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr