4 ARNAB AND TORR: BOTTOM-UP INSTANCE SEGMENTATION
Pixelwise CNN  Higher Order CRF  
Instance Identification  Instance CRF  
K+1  
Object Detector  
D+1  End-to-end object segmentation subnetwork  
End-to-end Instance segmentation subnetwork  
Input  
Output  Object  
Segmentation  
Identified  
Instances  
Figure 2: Overview of our end-to-end method. Our system ﬁrst computes a category-level
semantic segmentation of the image. A CRF with higher-order detection potentials is used
to obtain this result in a semantic segmentation subnetwork. This results in a W×H×(K+
1)dimensional volume where Kis the number of foreground classes. WandHare the
image’s width and height respectively. The original detections and the detector conﬁdences
recalibrated by the Higher Order CRF are used to identify instances in the image, producing
aW×H×(D+1)dimensional volume where Dis the number of detections (variable per
image). Thereafter, another Instance CRF is used to compute the ﬁnal result.
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr