capture more context, modelling co-occurrence of object classes [18 –20], and utilising
the results of object detectors [6, 21, 22].
Recent advances in deep learning have allowed us to replace hand-crafted features
with features learned speciﬁcally for semantic segmentation. The strength of these
representations was illustrated by [3] who achieved signiﬁcant improvements over
previous hand-crafted methods without using any CRF post-processing. Chen et al. [12]
showed further improvements by post-processing the results of a CNN with a CRF.
Subsequent works [9 –11, 23] have taken this idea further by incorporating a CRF as
layers within a deep network and then learning parameters of both the CRF and CNN
together via backpropagation.
In terms of enhancements to conventional CRF models, Ladicky et al. [6] proposed using an off-the-shelf object detector to provide additional cues for semantic
segmentation. Unlike other approaches that reﬁne a bounding-box detection to produce
a segmentation [8, 24], this method used detector outputs as a soft constraint and can
thus recover from object detection errors. Their formulation, however, used graph-cut
inference, which was only tractable due to the absence of dense pairwise potentials.
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr