by [29] where a Recurrent Neural Network outputs an object instance at each time step. This
method, however, has not been evaluated on multiple classes.
Our method also ﬁrst performs a bottom-up, semantic segmentation of the image. However, our semantic segmentation network uses the outputs of an object detector as an additional cue in its ﬁnal CRF layer. During inference, each object hypothesis from the detector
is evaluated in light of other energies in the CRF. Thus, the relative score of false positive
detections can be decreased, and correct detections increased. These rescored detections
are then used to identify instances from the initial semantic segmentation. The fact that our
system combines information from semantic segmentation and object detection networks allows us to bypass the complexity of the “instance bounding box” and “instance number”
prediction steps used by [24].
3 Proposed Approach
Our proposed method ﬁrst performs a semantic segmentation of the input image, classifying
each pixel into one of K+1 categories where Kis the number of foreground classes. The
resulting semantic segmentation is then reﬁned into an instance-level segmentation, where
4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr