Structured Prediction Theory Based on Factor Graph Complexity
We present a general theoretical analysis of structured prediction with a series of new results. We give new data-dependent margin guarantees for structured prediction for a very wide family of loss functions and a general family of hypotheses, with an arbitrary factor graph decomposition. These are the tightest margin bounds known for both standard multi-class and general structured prediction problems. Our guarantees are expressed in terms of a data-dependent complexity measure, factor graph complexity, which we show can be estimated from data and bounded in terms of familiar quantities for several commonly used hypothesis sets along with a sparsity measure for features and graphs. Our proof techniques include generalizations of Talagrand's contraction lemma that can be of independent interest. We further extend our theory by leveraging the principle of Voted Risk Minimization (VRM) and show that learning is possible even with complex factor graphs. We present new learning bounds for this advanced setting, which we use to design two new algorithms, Voted Conditional Random Field (VCRF) and Voted Structured Boosting (StructBoost). These algorithms can make use of complex features and factor graphs and yet benefit from favorable learning guarantees. We also report the results of experiments with VCRF on several datasets to validate our theory.
This paper studies generalization bounds for structured prediction. In line with previous work, the authors adopt a structured prediction model based on a scoring function. In particular they allow for such function to factorize in terms of scoring function that account only for a smaller subset of the output coordinates. A variation of the classical Rademacher complexity is proposed, based on this formulation and generalization bounds on the generalization error are discussed. The authors draw connections with previous work and then present an optimization algorithm to approach structured prediction problems.
The paper is well written and motivated. In particular the problem considered is relevant. On the downside there are some issues related to the interpretability of the presented results:  - In Theorem 1 the generalization error is bounded in terms of the additive or multiplicative empirical margin losses. However their formulation at Eq. (5) and (6) is hard to interpret and would benefit from a comment. This is problematic since it is not clear how these quantities are related to the algorithmic approaches discussed in Sec. 5. In particular it is not clear how the approaches described in Sec. 5 are related to minimizing the additive or multiplicative empirical margin loss.  [EDIT]: I thank the authors' for clarifying the role of the additive and multiplicative empirical margin loss in perspective of the algorithms proposed in Sec. 5. I feel the   - Theorem 2 is quite hard to parse. In particular the role of sparsity factor s and its dependence on the maximum cardinality of each factor is not clear and deserves a deeper analysis. Indeed, in the worse case scenario it is bounded by m max_i |F_i| d_i, but in the case of NLP described in Line 192, max_i |F_i| = +\infty if I understood correctly the model (since sentences can have arbitrary length). This means that in the case of infinite dimensional output spaces only factor graphs with uniformly bounded cardinality can be used. This limit the applicability of the result and should be discussed.  [EDIT]: the authors reply did not clarify my doubts. In particular, when Y is infinite: 1) why a factor graph F should have finite cardinality? and 2) the dependency of max_i |F_i| from the number of examples "m" could dramatically affect the bound.   However, most structured prediction settings assume Y of finite cardinality so this is a second-order issue.  - Section 3.3 draws connection with previous work. While it is interesting and relevant to understand how previous results can be cast in the setting considered, the discussion is difficult to follow. For instance, in the case of the multi-class classification the paper does not provide a direct formulation of the problem within their notation and therefore it is not clear how to interpret the theoretical results in this setting. Moreover, to connect to previous work the authors use results (e.g. Lemma 6), which are reported in the appendices and not directly connected to the main results of this paper (Theorem 1 and 2). This makes the discussion harder to follow.  - Sec. 5: as mentioned above, one problem of the discussion in Sec. 5 is that it lacks a direct connection with the theoretical results presented in this work. A further issue is that the authors do not comment on the relation between the ideal empirical (non-convex) problem and the convex surrogates that are reviewed/proposed. In particular it would be interesting to understand how tight the relation of the VCRF method is with specific losses. For instance in the case of multi-class classification it is known that StructSVM - which belongs to the general framework proposed in Eq. (9) - is not consistent.   Minor notes:  - Line 158: the authors state that the empirical factor graph Rademacher complexity is sharply concentrated around its expectation but I was not able to find a proof of this in the supplementary material.   - Line 168: "linear functions", did the authors mean "linear spaces"?  - Line 296 - 297: the authors used \delta \psi(x,y,y') which was never defined.