image). Thereafter, another Instance CRF is used to compute the ﬁnal result.
the object class of each instance segment is obtained from the previous semantic segmentation. Both of these stages, while conceptually different, are fully differentiable and the entire
system can be implemented as a neural network.
Figure 2 shows our category-level segmentation network, consisting of a pixelwise CNN
[27] followed by a fully differentiable CRF with object detection potentials, as described in
Section 3.2. The object detection results that our CRF takes in, as well as the recalibrated
detection scores that it outputs, allows us to identify object instances, and obtain a ﬁnal
instance-level segmentation in the next stage of our pipeline. This is described in Section
3.3. However, we ﬁrst review Conditional Random Fields and introduce the notation used in
this paper.
3.1 Conditional Random Fields
Assume an image Iwith Npixels, indexed 1 ,2...N, and deﬁne a set of random variables,
X1,X2,...,XN, one for every pixel. In a generic labelling problem, we assign every pixel a
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr