instances, but can also handle multiple object classes.
Liang et al. [33] developed another proposal-free method
based on the semantic segmentation network of [ 6]. The
category-level segmentation, along with CNN features, was
used to predict instance-level bounding boxes. The number
of instances of each class was also predicted to enable a ﬁnal
spectral clustering step. However, this additional information predicted by Liang’s network could have been obtained
 Semantic Segmentation subnetwork  Instance Segmentation subnetwork Instance Unary Potentials Pixelwise 
CNN Semantic CRF Detector Box 
Global 
Shape Instance CRF 
Input Image Instance Segmentation 
Semantic Segmentation End-to-end Network 
K+1 
D+1 
Figure 2: Network overview: Our end-to-end trained network consists of semantic- and instance-segmentation modules. The
intermediate category-level segmentation, along with the outputs of an object detector, are used to reason about instances.
This is done by instance unary terms which use information from the detector’s bounding boxes, the initial semantic segmentation and also the object’s shape. A ﬁnal CRF is used to combine all this information together to obtain an instance
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.
Anurag Arnab Philip H. S Torr