training our Higher Order CRF, we use the publicly available Faster-RCNN object detection
framework [28]. The semantic segmentation performance on the VOC validation set is a
mean IoU of 73.4% when using only pairwise potentials in the CRF. This rises to 75.3%
when using detection potentials as well. The authors of [2] observed a similar increase when
using detection potentials.
4.2 Effect of detection potentials
To analyse the effect of our detection potentials, we evaluate our system by disabling the
higher order potentials in our Semantic Segmentation CRF. Because our detection potentials
are disabled, we can no longer use the output of the latent Yvariables as our recalibrated
detection score, and use the original detector’s conﬁdence score instead. As shown in Table
1, the result is that the APrat 0.5 is 3.7% lower and the APr
volis 3.1% lower.
The detection potentials improve instance segmentation performance as they improve
both our initial semantic segmentation, and also recalibrate the detector’s initial scores. To
decouple these two effects, we then include detection potentials in our Segmentation CRF,
but ignore the latent Yvariables output by the CRF and use the detector’s original conﬁdence
Traditional Scene Understanding problems such as Object Detection and
Semantic Segmentation have made breakthroughs in recent years due to the
adoption of deep learning. However, the former task is not able to localise
objects at a pixel level, and the latter task has no notion of different
instances of objects of the same class. We focus on the task of Instance
Segmentation which recognises and localises objects down to a pixel level. Our
model is based on a deep neural network trained for semantic segmentation. This
network incorporates a Conditional Random Field with end-to-end trainable
higher order potentials based on object detector outputs. This allows us to
reason about instances from an initial, category-level semantic segmentation.
Our simple method effectively leverages the great progress recently made in
semantic segmentation and object detection. The accurate instance-level
segmentations that our network produces is reflected by the considerable
improvements obtained over previous work.
Anurag Arnab Philip H. S. Torr