[19,20,12,37], our network requires no additional postprocessing to create an instance segmentation map as in
Fig. 1(c) and reasons about the entire image, rather than
independent proposals. This global reasoning allows our
method to produce more accurate segmentations. Our proposed system also handles a variable number of instances
per image, and thus does not assume a maximum number
of instances like [ 57,58].
3. Proposed Approach
Our network (Fig. 2) contains an initial semantic segmentation module. We use the semantic segmentation result, along with the outputs of an object detector, to compute
the unary potentials of a Conditional Random Field (CRF)
deﬁned over object instances. We perform mean ﬁeld inference in this random ﬁeld to obtain the Maximum a Posteriori (MAP) estimate, which is our labelling. Although ournetwork consists of two conceptually different parts – a semantic segmentation module, and an instance segmentation
network – the entire pipeline is fully differentiable, given
object detections, and trained end-to-end.
3.1. Semantic Segmentation subnetwork
Semantic Segmentation assigns each pixel in an image a
semantic class label from a given set, L. In our case, this
module uses the FCN8s architecture [ 38] which is based
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.
Anurag Arnab Philip H. S Torr