another source of imagery, professional photos on the interior design website Houzz (houzz.com). Our motivation for
using this different source of data was that, despite Houzz
photos being more “staged” (relative to Flickr photos), they
actually represent a larger variety of materials. For instance,
Houzz photos contain a wide range of types of polished
stone. With these sources of image data, we now describe
how we gather material annotations.
3.2. Segments, Clicks, and Patches
What speciﬁc kinds of material annotations make for a
good database? How should we collect these annotations?
The type of annotations to collect is guided in large part by
the tasks we wish to generate training data for. For some
tasks such as scene recognition, whole-image labels can
sufﬁce [ 31,34]. For object detection, labeled bounding
boxes as in PASCAL are often used [ 5]. For segmentation or
scene parsing tasks, per-pixel segmentations are required [ 22,
8]. Each style of annotation comes with a cost proportional
to its complexity. For materials, we decided to focus on two
problems, guided by prior work:
Patch material classiﬁcation. Given an image patch,
what kind of material is it at the center?
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.
Sean Bell Paul Upchurch Noah Snavely Kavita Bala