them into fully convolutional networks that take input of
any size and output classiﬁcation maps. This transformation
`tabby cat" `
96256384384256409640961000
96384256409640961000
256384tabby cat heatmapconvolutionalizationFigure 2. Transforming fully connected layers into convolution
layers enables a classiﬁcation net to output a heatmap. Adding
layers and a spatial loss (as in Figure 1) produces an efﬁcient machine for end-to-end dense learning.
is illustrated in Figure 2. (By contrast, nonconvolutional
nets, such as the one by Le et al. [20], lack this capability.)
Furthermore, while the resulting maps are equivalent to
the evaluation of the original net on particular input patches,
the computation is highly amortized over the overlapping
regions of those patches. For example, while AlexNet takes
1:2ms (on a typical GPU) to produce the classiﬁcation
scores of a 227227image, the fully convolutional version takes 22ms to produce a 1010grid of outputs from
a500500image, which is more than 5times faster than
the na ¨ıve approach1.
Convolutional networks are powerful visual models that yield hierarchies of
features. We show that convolutional networks by themselves, trained
end-to-end, pixels-to-pixels, exceed the state-of-the-art in semantic
segmentation. Our key insight is to build "fully convolutional" networks that
take input of arbitrary size and produce correspondingly-sized output with
efficient inference and learning. We define and detail the space of fully
convolutional networks, explain their application to spatially dense prediction
tasks, and draw connections to prior models. We adapt contemporary
classification networks (AlexNet, the VGG net, and GoogLeNet) into fully
convolutional networks and transfer their learned representations by
fine-tuning to the segmentation task. We then define a novel architecture that
combines semantic information from a deep, coarse layer with appearance
information from a shallow, fine layer to produce accurate and detailed
segmentations. Our fully convolutional network achieves state-of-the-art
segmentation of PASCAL VOC (20% relative improvement to 62.2% mean IU on 2012),
NYUDv2, and SIFT Flow, while inference takes one third of a second for a
typical image.
Jonathan Long Evan Shelhamer Trevor Darrell