of learning to predict an energy landscape such that each
basin corresponds to a single instance, while all ridges are
at the same height in the energy domain. As a consequence,
the watershed cut corresponds to a single threshold in the
energy, which does not lead to over segmentation. We refer
the reader to the lower half of Fig. 2 for an illustration of
the desired energy.
Unfortunately, learning the energy landscape from
scratch is a complex task. Therefore, we aid the network
by deﬁning an intermediate task, where we learn the direction of descent of the watershed energy. This is then passedthrough another set of network layers to learn the ﬁnal energy. In principle, one can interpret this network as learning
to perform the distance transform of each point within an
object instance to the instance’s boundary. Fig. 4 shows an
example of the input, intermediate results, and ﬁnal output
of our method. We refer the reader to Fig. 3 for an illustration of our network architecture.
4.1. Direction Network (DN)
Our approach leverages semantic segmentation as input
to focus only on relevant areas of the image. Note that our
network is agnostic to the choice of segmentation algorithm.
In our paper, we use the semantic segmentation results from
PSPNet [34].
Most contemporary approaches to instance segmentation use complex pipelines
involving conditional random fields, recurrent neural networks, object
proposals, or template matching schemes. In our paper, we present a simple yet
powerful end-to-end convolutional neural network to tackle this task. Our
approach combines intuitions from the classical watershed transform and modern
deep learning to produce an energy map of the image where object instances are
unambiguously represented as basins in the energy map. We then perform a cut at
a single energy level to directly yield connected components corresponding to
object instances. Our model more than doubles the performance of the
state-of-the-art on the challenging Cityscapes Instance Level Segmentation
task.
Min Bai Raquel Urtasun