inference in a graphical model with only unary potentials. However, for arbitrary graphical models
it is NP-hard to ﬁnd the maximizing conﬁguration ysince the inference program generally requires
a search over a space of sizeQN
i=1jYij. Note also that log-linear models are a special case of this
program, with F(x;y;w) =w>(x;y)and(x;y)denoting a feature vector computed using the
input-output pair (x;y).
In this work, we consider the general setting where F(x;y;w)is an arbitrary scalar-valued function ofwand(x;y). In our experiments Fis a function composition of non-linear base mappings
like convolutions, rectiﬁcations, pooling etc. We let the probability of an arbitrary conﬁguration ^y
be given by the annealed soft-max p(x;y)(^yjw;) =1
Z(x;w)exp(F(x;^y;w))1=:HerebyZ(x;w)
refers to the partition function, normalizing the distribution p(x;y)to lie within the probability simplexviaZ(x;w) =P
Many problems in real-world applications involve predicting several random
variables which are statistically related. Markov random fields (MRFs) are a
great mathematical tool to encode such relationships. The goal of this paper is
to combine MRFs with deep learning algorithms to estimate complex
representations while taking into account the dependencies between the output
random variables. Towards this goal, we propose a training algorithm that is
able to learn structured models jointly with deep features that form the MRF
potentials. Our approach is efficient as it blends learning and inference and
makes use of GPU acceleration. We demonstrate the effectiveness of our
algorithm in the tasks of predicting words from noisy images, as well as
multi-class classification of Flickr photographs. We show that joint learning
of the deep features and the MRF parameters results in significant performance
gains.
Liang-Chieh Chen Alexander G. Schwing Alan L. Yuille Raquel Urtasun