method, the mean IoU increases from 72.9% to 73.6%, which is signiﬁcantly less than
the 74.9% which we obtained using only our detection potentials without superpixels
(Table 1).
Our detection potentials perform better since our latent Ydetection variables model
whether the detection hypothesis is accepted or not. Our CRF inference is able to evaluate
object detection inputs in light of other potentials. Inference increases the relative score
of detections which agree with the segmentation, and decreases the score of detections
which do not agree with other energies in the CRF. Figures 2 b) and d) show examples
of false-positive detections that have been ignored and correct detections that have been
used to reﬁne our segmentation. Our baseline, on the other hand, is far more sensitive to
erroneous detections as it cannot adjust the weight given to them during inference.
7 Conclusion
We presented a CRF model with two different higher order potentials to tackle the
semantic segmentation problem. The ﬁrst potential is based on the intuitive idea that
object detection can provide useful cues for semantic segmentation. Our formulation is
capable of automatically rejecting false object detections that do not agree at all with
the semantic segmentation. Secondly, we used a potential that encourages superpixels to
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr