not been annotated in the dataset, but both our initial semantic segmentation and object detection networks have identiﬁed it.
Note that the ﬁrst three images are from the VOC dataset, and the last three from SBD. Annotations in the VOC dataset are
more detailed, and also make more use of the grey “ignore” label to indicate uncertain areas in the image. The ﬁrst column
shows the input image, and the results of our object detector which are another input to our network. Best viewed in colour.
Input image Semantic Segmentation Instance Segmentation Ground truth
Figure 8: Failure cases of our method. First row: Both our initial detector, and semantic segmentation system did not
identify a car in the background. Additionally, the “brown” person prediction actually consists of two people that have been
merged together. This is because the detector did not ﬁnd the background person. Second row: Our initial semantic segmentation identiﬁed the table, but it is not there in the Instance Segmentation. This is because there was no “table detection” to
associate these pixels with. Using heuristics, we could propose additional detections in cases like these. However, we have
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.
Anurag Arnab Philip H. S Torr