The edges in our bipartite graph connect ground-truth and
predicted segments. The edge weights are given by the
IoU between the ground truth and predicted segments if
they share the same semantic class label, and zero otherwise. Leftover segments are matched to “dummy” nodes
with zero overlap.
Additionally, the ordering of the instances in our network
are actually determined by the object detector, which remains static during training. As a result, the ordering of
our predictions does not ﬂuctuate much during training – it
only changes in cases where there are multiple detections
overlapping an object.
3.5. Network Training
We ﬁrst train a network for semantic segmentation with
the standard cross-entropy loss. In our case, this network is
FCN8s [ 38] with a CRF whose inference is unrolled as an
RNN and trained end-to-end, as described in [ 60] and [ 2].
To this pretrained network, we append our instance segmentation subnetwork, and ﬁnetune with instance segmentation
annotations and only the loss detailed in Sec. 3.4. For the
semantic segmentation subnetwork, we train with an initial
learning rate of 10−8, momentum of 0.9 and batch size of
20. The learning rate is low since we do not normalise the
Semantic segmentation and object detection research have recently achieved
rapid progress. However, the former task has no notion of different instances
of the same object, and the latter operates at a coarse, bounding-box level. We
propose an Instance Segmentation system that produces a segmentation map where
each pixel is assigned an object class and instance identity label. Most
approaches adapt object detectors to produce segments instead of boxes. In
contrast, our method is based on an initial semantic segmentation module, which
feeds into an instance subnetwork. This subnetwork uses the initial
category-level segmentation, along with cues from the output of an object
detector, within an end-to-end CRF to predict instances. This part of our model
is dynamically instantiated to produce a variable number of instances per
image. Our end-to-end approach requires no post-processing and considers the
image holistically, instead of processing independent proposals. Therefore,
unlike some related work, a pixel cannot belong to multiple instances.
Furthermore, far more precise segmentations are achieved, as shown by our
state-of-the-art results (particularly at high IoU thresholds) on the Pascal
VOC and Cityscapes datasets.
Anurag Arnab Philip H. S Torr