performing dense classiﬁcation over the image with a fully
connected conditional random ﬁeld (CRF) model [ 12]. By
replacing the fully connected layers of the CNN with convolutional layers [ 24], the computational burden is signiﬁcantly
lower than a naive sliding window approach.
In summary, we make two new contributions:
We introduce a new material dataset, MINC , and 3stage crowdsourcing pipeline for efﬁciently collecting
millions of click labels (Section 3.2).
Our new semantic segmentation method combines a
fully-connected CRF with unary predictions based on
CNN learned features (Section 4.2) for simultaneous
material recognition and segmentation.
2. Prior Work
Material Databases. Much of the early work on material
recognition focused on classifying speciﬁc instances of textures or material samples. For instance, the CUReT [ 4]
database contains 61 material samples, each captured under
205 different lighting and viewing conditions. This led to
research on the task of instance-level texture or material classiﬁcation [ 15,30], and an appreciation of the challenges of
building features that are invariant to pose and illumination.
Recognizing materials in real-world images is a challenging task. Real-world
materials have rich surface texture, geometry, lighting conditions, and
clutter, which combine to make the problem particularly difficult. In this
paper, we introduce a new, large-scale, open dataset of materials in the wild,
the Materials in Context Database (MINC), and combine this dataset with deep
learning to achieve material recognition and segmentation of images in the
wild.
  MINC is an order of magnitude larger than previous material databases, while
being more diverse and well-sampled across its 23 categories. Using MINC, we
train convolutional neural networks (CNNs) for two tasks: classifying materials
from patches, and simultaneous material recognition and segmentation in full
images. For patch-based classification on MINC we found that the best
performing CNN architectures can achieve 85.2% mean class accuracy. We convert
these trained CNN classifiers into an efficient fully convolutional framework
combined with a fully connected conditional random field (CRF) to predict the
material at every pixel in an image, achieving 73.1% mean class accuracy. Our
experiments demonstrate that having a large, well-sampled dataset such as MINC
is crucial for real-world material recognition and segmentation.
Sean Bell Paul Upchurch Noah Snavely Kavita Bala