We achieve state-of-the-art segmentation performance on the PASCAL VOC
benchmark with these trainable higher order potentials.
Keywords: Semantic Segmentation, Conditional Random Fields, Deep Learning,
Convolutional Neural Networks
1 Introduction
Semantic segmentation involves assigning a visual object class label to every pixel in an
image, resulting in a segmentation with a semantic meaning for each segment. While a
strong pixel-level classiﬁer is critical for obtaining high accuracy in this task, it is also
important to enforce the consistency of the semantic segmentation output with visual
features of the image. For example, segmentation boundaries should usually coincide
with strong edges in the image, and regions in the image with similar appearance should
have the same label.
Recent advances in deep learning have enabled researchers to create stronger classiﬁers, with automatically learned features, within a Convolutional Neural Network
(CNN) [1 –3]. This has resulted in large improvements in semantic segmentation accuracy on widely used benchmarks such as PASCAL VOC [4]. CNN classiﬁers are now
considered the standard choice for pixel-level classiﬁers used in semantic segmentation.
On the other hand, probabilistic graphical models have long been popular for structured prediction of labels, with constraints enforcing label consistency. Conditional
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr