While fully convolutionalized classiﬁers can be ﬁnetuned to segmentation as shown in 4.1, and even score
highly on the standard metric, their output is dissatisfyingly
coarse (see Figure 4). The 32 pixel stride at the ﬁnal prediction layer limits the scale of detail in the upsampled output.
We address this by adding links that combine the ﬁnal
prediction layer with lower layers with ﬁner strides. This
3Using the publicly available CaffeNet reference model.
4Since there is no publicly available version of GoogLeNet, we use
our own reimplementation. Our version is trained with less extensive data
augmentation, and gets 68.5% top-1 and 88.4% top-5 ILSVRC accuracy.
5Using the publicly available version from the Caffe model zoo.Table 1. We adapt and extend three classiﬁcation convnets to segmentation. We compare performance by mean intersection over
union on the validation set of PASCAL VOC 2011 and by inference time (averaged over 20 trials for a 500500 input on an
NVIDIA Tesla K40c). We detail the architecture of the adapted
nets as regards dense prediction: number of parameter layers, receptive ﬁeld size of output units, and the coarsest stride within the
Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.
Jonathan Long Evan Shelhamer Trevor Darrell