4 Arnab et al.
share boundaries with objects in an image, which is not always true. As a result, several
authors [5, 7] have employed higher order potentials deﬁned over superpixels that encourage label consistency over regions, but do not strictly enforce it. This approach also
allows multiple, non-hierarchical layers of superpixels to be integrated. Our formulation
uses this kind of higher order potential, but in an end-to-end trainable CNN.
Graphical models have been used with CNNs in other areas besides semantic segmentation, such as in pose-estimation [29] and group activity recognition [30]. Alternatively,
Ionescu et al. [31] incorporated structure into a deep network with structured matrix
layers and matrix backpropagation. However, the nature of models used in these works
is substantially different to ours. Some early works that advocated gradient backpropagation through graphical model inference for parameter optimisation include [32, 33]
and [34].
Our work differentiates from the above works since, to our knowledge, we are the ﬁrst
to propose and conduct a thorough experimental investigation of higher order potentials
that are based on detection outputs and superpixel segmentation in a CRF which is
learned end-to-end in a deep network. Note that although [7] formulated mean ﬁeld
We address the problem of semantic segmentation using deep learning. Most
segmentation systems include a Conditional Random Field (CRF) to produce a
structured output that is consistent with the image's visual features. Recent
deep learning approaches have incorporated CRFs into Convolutional Neural
Networks (CNNs), with some even training the CRF end-to-end with the rest of
the network. However, these approaches have not employed higher order
potentials, which have previously been shown to significantly improve
segmentation performance. In this paper, we demonstrate that two types of
higher order potential, based on object detections and superpixels, can be
included in a CRF embedded within a deep network. We design these higher order
potentials to allow inference with the differentiable mean field algorithm. As
a result, all the parameters of our richer CRF model can be learned end-to-end
with our pixelwise CNN classifier. We achieve state-of-the-art segmentation
performance on the PASCAL VOC benchmark with these trainable higher order
potentials.
Anurag Arnab Sadeep Jayasumana Shuai Zheng Philip Torr