this fusion with learning only from the pool4 layer (which
resulted in poor performance), and simply decreasing the
learning rate without adding the extra link (which results
in an insigniﬁcant performance improvement, without improving the quality of the output).
We continue in this fashion by fusing predictions from
pool3 with a 2upsampling of predictions fused from
pool4 andconv7 , building the net FCN-8s. We obtain
6Max fusion made learning difﬁcult due to gradient switching.
FCN-32s FCN-16s FCN-8s Ground truth
Figure 4. Reﬁning fully convolutional nets by fusing information
from layers with different strides improves segmentation detail.
The ﬁrst three images show the output from our 32, 16, and 8
pixel stride nets (see Figure 3).
Table 2. Comparison of skip FCNs on a subset of PASCAL
VOC2011 validation7. Learning is end-to-end, except for FCN32s-ﬁxed, where only the last layer is ﬁne-tuned. Note that FCN32s is FCN-VGG16, renamed to highlight stride.
pixel
acc.mean
acc.mean
IUf.w.
IU
Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.
Jonathan Long Evan Shelhamer Trevor Darrell