semantic segmentation literature which always have a ﬁxed number of labels for all images
in the dataset [17, 18, 35, 37]. Zheng et al. [37] showed how the iterative mean ﬁeld inference algorithm can be unrolled into a series of differentiable operations and be implemented
as a recurrent neural network. We employ a similar method so that our dynamic CRF is fully
differentiable and can thus be incorporated into a neural network. This is achieved by using
CRF weights that are not class-speciﬁc, and can thus generalise to any number of labels per
image. A label here refers to an instance associated with a detection and has no semantic
meaning. For example, in Fig. 1, label “1” was associated with a dining table, whilst in Fig.
2 it is a sheep. It therefore does not make sense to have class-speciﬁc weights in any case.
This approach to instance segmentation is bottom-up, in the sense that we ﬁrst perform
category-level segmentation of each pixel before predicting instances. However, the fact
that we used object detection cues in performing our semantic segmentation helps us to
subsequently reason about instances. Moreover, the fact that we used results from both
semantic segmentation and object detection makes sense since instance segmentation can be
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr