manually curated segments with low conﬁdence, giving a
total of 72K shapes. To better balance the categories, we
manually segmented a few hundred extra samples for sky,
foliage andwater .
Since some of the OpenSurfaces categories are difﬁcult
for humans, we consolidated these categories. We found
that many AMT workers could not disambiguate stone from
concrete ,clear plastic from opaque plastic , and granite
from marble . Therefore, we merged these into stone ,plastic ,
andpolished stone respectively. Without this merging, many
ground truth examples in these categories would be incorrect.
The ﬁnal list of 23 categories is shown in Table 1. The
category other is different in that it was created by combining
various smaller categories.
Clicks. Since we want to expand our dataset to millions
of samples, we decided to augment OpenSurfaces segments
by collecting clicks : single points in an image along with a
material label, which are much cheaper and faster to collect.
Figure 3 shows our pipeline for collecting clicks.
Initially, we tried asking workers to click on examples
of a given material in a photo. However, we found that
workers would get frustrated if the material was absent in
too many of the photos. Thus, we added an initial ﬁrst
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.
Sean Bell Paul Upchurch Noah Snavely Kavita Bala