On Structured Prediction Theory with Calibrated Convex Surrogate Losses
We provide novel theoretical insights on structured prediction in the context of efficient convex surrogate loss minimization with consistency guarantees. For any task loss, we construct a convex surrogate that can be optimized via stochastic gradient descent and we prove tight bounds on the so-called "calibration function" relating the excess surrogate risk to the actual risk. In contrast to prior related work, we carefully monitor the effect of the exponential number of classes in the learning guarantees as well as on the optimization complexity. As an interesting consequence, we formalize the intuition that some task losses make learning harder than others, and that the classical 0-1 loss is ill-suited for structured prediction.
class loss functions [3,39,48]. A large body of work specifically focuses on the related tasks of ranking [18,9,40] and ordinal regression [37]. Contributions. In this paper, we study consistent convex surrogate losses specifically in the context of an exponential number of classes. We argue that even while being consistent, a convex surrogate might not allow efficient learning. As a concrete example, Ciliberto et al. [10] recently proposed a consistent approach to structured prediction, but the constant in their generalization error bound can be exponentially large as we explain in Section 5. There are two possible sources of difficulties from the optimization perspective: to reach adequate accuracy on the task loss, one might need to optimize a surrogate loss to exponentially small accuracy; or to reach adequate accuracy on the surrogate loss, one might need an exponential number of algorithm steps because of exponentially large constants in the convergence rate. We propose a theoretical framework that jointly tackles these two aspects and allows to judge the feasibility of efficient learning. In particular, we construct a calibration function [43], i.e., a function setting the relationship between accuracy on the surrogate and task losses, and normalize it by the means of convergence rate of an optimization algorithm.
Aiming for the simplest possible application of our framework, we propose a family of convex surrogates that are consistent for any given task loss and can be optimized using stochastic gradient descent. For a special case of our family (quadratic surrogate), we provide a complete analysis including general lower and upper bounds on the calibration function for any task loss, with exact values for the 0-1, block 0-1 and Hamming losses. We observe that to have a tractable learning algorithm, one needs both a structured loss (not the 0-1 loss) and appropriate constraints on the predictor, e.g., in the form of linear constraints for the score vector functions. Our framework also indicates that in some cases it might be beneficial to use non-consistent surrogates. In particular, a non-consistent surrogate might allow optimization only up to specific accuracy, but exponentially faster than a consistent one.
We introduce the structured prediction setting suitable for studying consistency in Sections 2 and 3. We analyze the calibration function for the quadratic surrogate loss in Section 4. We review the related works in Section 5 and conclude in Section 6.
The paper examines consistency of surrogate losses for multiclass prediction. The authors present their results using the formalism of structured prediction. Alas, there is no direct connection or exploitation of the separability of structured prediction losses. The paper is overly notated and variables are frequently overloaded. I got the feeling that the authors are trying to look mathematically fancy at the expense of readability. As for the technical content, I find the contribution in terms of new proof techniques quite marginal. The authors build on results by Toingo(Tong + Ingo) and the theorems seem lackluster to me.
From algorithmic prospective the paper has rather little to offer. The quadratic loss is either intractable to calculate or in the case of separable it amounts to sum of squares over the vector L_j. There's no attempt to experimentally validate the calibrated losses. Also absent is a sincere discussion on the importance of calibration given the empirical success of overly-specified models.
In "Contributions" paragraph the word "exponential" repeats over and over. It would not be clear to a reader who is not familiar with the setting what is the notion of exponential dependency.
The sentence "...  sets of possible predictions and correct outputs always coincide and do not depend on x." left me baffled. 
There is excessive overload of notation for L which is used as a function, specific losses, table, etc...
minimal -> smallest / lowest
We denote the y-th component of f with f y
"... for now, we assume that a family of score functions F consists of all vector-valued Borel measurable functions f : X → F where F ⊆ R k ..." and then
what ?
Is it feasible to find inf φ ?
"... equals the smallest excess of the conditional surrogate risk when the excess of the conditional actual risk is larger than ..." left me bewildered again.
I feel that paper would have rather little appeal at NIPS and a natural venue would be COLT or ALT.v