sliding window approach to localize patch classiﬁcation of
objects. We build on this body of work in deep learning to
solve our problem of material recognition and segmentation.
2
Brick Carpet Ceramic Fabric Foliage Food Glass Hair
Leather Metal Mirror Other Painted Paper Plastic Pol. stone
Skin Sky Stone Tile Wallpaper Water Wood
Figure 2. Example patches from all 23 categories of the Materials in Context Database (MINC ). Note that we sample patches so that the
patch center is the material in question (and not necessarily the entire patch). See Table 1 for the size of each category.
3. The Materials in Context Database (MINC)
We now describe the methodology that went into building
our new material database. Why a new database? We needed
a dataset with the following properties:
Size: It should be sufﬁciently large that learning methods can generalize beyond the training set.
Well-sampled : Rare categories should be represented
with a large number of examples.
Diversity : Images should span a wide range of appearances of each material in real-world settings.
Number of categories : It should contain many different materials found in the real world.
3.1. Sources of data
We decided to start with the public, crowdsourced
OpenSurfaces dataset [ 1] as the seed for MINC since it is
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.
Sean Bell Paul Upchurch Noah Snavely Kavita Bala