network is agnostic to the choice of segmentation algorithm.
In our paper, we use the semantic segmentation results from
PSPNet [34].
The network takes as input the original RGB image gated
by a binarized semantic segmentation, where all pixels that
are not part of one of the semantic classes of interest are set
to zero. The input image is augmented by adding the semantic segmentation as a fourth channel. Because the RGB
values range from 0 to 255 for each channel before mean
subtraction, we likewise scale the encoding of the semantic
segmentation image such that the class labels are equally
spaced numerically, and the variance is approximately equal
to that of values in the RGB channels.
To aid the model in producing an accurate energy landscape, we pre-train the overall network’s ﬁrst part (referenced here as the Direction Network, DN) to estimate the
direction of descent of the energy at each pixel. We parameterize it with a unit vector pointing away from the nearest
point on the object’s instance boundary. This supervision
gives a very strong training signal: the direction of the nearest boundary is directly encoded in the unit vector. Furthermore, a pair of pixels straddling an occlusion boundary
between two objects will have target unit vectors pointing
in opposite directions. Here, associating a pixel with the
Most contemporary approaches to instance segmentation use complex pipelines
involving conditional random fields, recurrent neural networks, object
proposals, or template matching schemes. In our paper, we present a simple yet
powerful end-to-end convolutional neural network to tackle this task. Our
approach combines intuitions from the classical watershed transform and modern
deep learning to produce an energy map of the image where object instances are
unambiguously represented as basins in the energy map. We then perform a cut at
a single energy level to directly yield connected components corresponding to
object instances. Our model more than doubles the performance of the
state-of-the-art on the challenging Cityscapes Instance Level Segmentation
task.
Min Bai Raquel Urtasun