bounding boxes, which are then further reﬁned to obtain instance level segmentation.
Deep structured models: [32, 33] ﬁrst use CNNs to perform local instance disambiguation and labelling, followed
by a global conditional random ﬁeld (CRF) to achieve instance label consistency. Recent work by [2] uses object
detection proposals in conjunction with a deep high order
CRF to reason about pixel assignment in overlapping object
proposal boxes.
Template matching: [28] extracts image features using
CNNs to assign a sector label to each pixel in an object instance, which corresponds to one of eight discretized radial
bins around the object’s visible center. A template matching
scheme is then used to associate instance center proposals
and pixels with an object instance.
Recurrent Networks: [24] uses CNNs for feature extraction, followed by a recurrent neural network (RNN) that
generates instance labels for one object at a time. The recurrent structures (based on ConvLSTM [25]) keep track of
instances that have already been generated, and inhibit these
regions from further instance generation. Additionally, [23]
extracts image features similar to [28] and employs a fairly
complex pipeline including a ConvLSTM structure to direct
Most contemporary approaches to instance segmentation use complex pipelines
involving conditional random fields, recurrent neural networks, object
proposals, or template matching schemes. In our paper, we present a simple yet
powerful end-to-end convolutional neural network to tackle this task. Our
approach combines intuitions from the classical watershed transform and modern
deep learning to produce an energy map of the image where object instances are
unambiguously represented as basins in the energy map. We then perform a cut at
a single energy level to directly yield connected components corresponding to
object instances. Our model more than doubles the performance of the
state-of-the-art on the challenging Cityscapes Instance Level Segmentation
task.
Min Bai Raquel Urtasun