segmentation. The output of the semantic segmentation module is a ﬁxed size W×H×(K+ 1) tensor where Kis the
number of object classes, excluding background, in the dataset. The ﬁnal output, however, is of a variable W×H×(D+ 1)
dimensions where Dis the number of detected objects (and one background label).
from an object detector. Arnab et al. [3] also started with an
initial semantic segmentation network [ 2], and combined
this with the outputs of an object detector using a CRF to
reason about instances. This method was not trained endto-end though, and could not really recover from errors in
bounding-box localisation or occlusion.
Our method also has an initial semantic segmentation
subnetwork, and uses the outputs of an object detector.
However, in contrast to [ 3] it is trained end-to-end to improve on both semantic- and instance-segmentation performance (to our knowledge, this is the ﬁrst work to achieve
this). Furthermore, it can handle detector localisation errors and occlusions better due to the energy terms in our
end-to-end CRF. In contrast to detection-based approaches
[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.
Anurag Arnab Philip H. S Torr