3.2 Higher Order Detection Potentials
In addition to the standard unary and densely-connected pairwise terms, we also include a
higher order term based on object detections. Intuitively, a state-of-the-art object detector
[11, 28] can help segmentation performance in cases where the detector correctly predicts an
object at a location in which the pixels have been misclassiﬁed by the unary classiﬁer, and
thus reduce the recognition error [6]. Arnab et al. [2] formulated object detection potentials
in a manner that is amenable to the differentiable mean ﬁeld inference algorithm so that it
could be integrated into a neural network, and showed that it does indeed improve semantic
segmentation performance. We review the detection potentials described in [2] as they also
help us to reason about instances in an input image, a problem not considered by [2].
Assume that we have Dobject detections for an input image (this number varies for every
image), and that the dthdetection is of the form (ld,sd,Fd,Bd)where ld∈Lis the object class
label of the detected object, sdis the detector’s conﬁdence score, Fd⊂{1,2,...N}is the set
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr