fully connected CRF and particularly effective in the context of semantic segmentation.
After the ﬁrst version of our manuscript was made publicly available, it came to our attention that
two other groups have independently and concurrently pursued a very similar direction, combining
DCNNs and densely connected CRFs (Bell et al., 2014; Zheng et al., 2015). There are several
differences in technical aspects of the respective models. Bell et al. (2014) focus on the problem
of material classiﬁcation, while Zheng et al. (2015) unroll the CRF mean-ﬁeld inference steps to
convert the whole system into an end-to-end trainable feed-forward network.
We have updated our proposed “DeepLab” system with much improved methods and results in our
latest work (Chen et al., 2016). We refer the interested reader to the paper for details.
3 C ONVOLUTIONAL NEURAL NETWORKS FOR DENSE IMAGE LABELING
Herein we describe how we have re-purposed and ﬁnetuned the publicly available Imagenetpretrained state-of-art 16-layer classiﬁcation network of (Simonyan & Zisserman, 2014) (VGG-16)
into an efﬁcient and effective dense feature extractor for our dense semantic image segmentation
Deep Convolutional Neural Networks (DCNNs) have recently shown state of the
art performance in high level vision tasks, such as image classification and
object detection. This work brings together methods from DCNNs and
probabilistic graphical models for addressing the task of pixel-level
classification (also called "semantic image segmentation"). We show that
responses at the final layer of DCNNs are not sufficiently localized for
accurate object segmentation. This is due to the very invariance properties
that make DCNNs good for high level tasks. We overcome this poor localization
property of deep networks by combining the responses at the final DCNN layer
with a fully connected Conditional Random Field (CRF). Qualitatively, our
"DeepLab" system is able to localize segment boundaries at a level of accuracy
which is beyond previous methods. Quantitatively, our method sets the new
state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching
71.6% IOU accuracy in the test set. We show how these results can be obtained
efficiently: Careful network re-purposing and a novel application of the 'hole'
algorithm from the wavelet community allow dense computation of neural net
responses at 8 frames per second on a modern GPU.
Liang-Chieh Chen George Papandreou Iasonas Kokkinos Kevin Murphy Alan L. Yuille