5
Figure 5. Varying patch scale . We train/test patches of different
scales (the patch locations do not vary). The optimum is a trade-off
between context and spatial resolution. CNN: AlexNet.
Architecture Validation Test
AlexNet [13] 82.2% 81.4%
GoogLeNet [28] 85.9% 85.2%
VGG-16 [27] 85.6% 84.8%
Table 2. Patch material classiﬁcation results. Mean class accuracy for different CNNs trained on MINC. See Section 5.1.
Sky 97.3% Food 90.4% Wallpaper 83.4% Glass 78.5%
Hair 95.3% Leather 88.2% Tile 82.7% Fabric 77.8%
Foliage 95.1% Other 87.9% Ceramic 82.7% Metal 77.7%
Skin 93.9% Pol. stone 85.8% Stone 82.7% Mirror 72.0%
Water 93.6% Brick 85.1% Paper 81.8% Plastic 70.9%
Carpet 91.6% Painted 84.2% Wood 81.3%
Table 3. Patch test accuracy by category. CNN: GoogLeNet . See
the supplemental material for a full confusion matrix.
5. Experiments and Results
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.
Sean Bell Paul Upchurch Noah Snavely Kavita Bala