Multilabel Structured Output Learning with Random Spanning Trees of Max-Margin Markov Networks
We show that the usual score function for conditional Markov networks can be written as the expectation over the scores of their spanning trees. We also show that a small random sample of these output trees can attain a significant fraction of the margin obtained by the complete graph and we provide conditions under which we can perform tractable inference. The experimental results confirm that practical learning is scalable to realistic datasets using this approach.
***** Update after author feedback at the end ********************************************************quality:  8 (out of 10)clarity: 7originality: 6significance: 7SUMMARY: The authors consider the use of random spanning trees to do structured prediction when the underlying graph structure on the labels is unknown. They motivate their approach by first showing that under some assumptions, the discriminant function over a fully connected graph on the labels can be expressed as the uniform expectation of the discriminant functions over random spanning trees. Through a sampling result, they then show that with high probability over samples of random spanning trees, there is a conical combination of these trees which achieve a substantial fraction of the margin of a predictor which uses the complete graph, and then prove a related risk bound for conical combination over random trees. This motivates to optimize the margin for conical combination of trees as predictors, and the author proposes a primal (and dual) formulation for this optimization problem (somewhat analog to the structured SVM), for which a standard dual subgradient method is proposed as in previous work. They then show that the maximizing joint label for the combination of trees (inference) can be done exactly (under an assumption that be checked at run-time) by looking through the K-best list for each spanning tree (the latter can be obtained by dynamic programming, as was already mentioned in Tsochantaridis et al. [JMLR 2005]). Experiments on standard multilabel datasets show a small improvement over alternative methods. Moreover, the authors investigate the proportion of times they can get an exact solution (according to the checkable condition they derived) as a function of K and the number of trees on the datasets. On some datasets, the exact solution is obtained a large fraction of time even for small K.EVALUATION:The authors claim in their introduction (line 051) "Our work is not based on any of these approaches, despite superficial resemblances to (for example) the trees in tree-based relaxations and the use of random trees in [5]." But I see this paper as a joint learning / joint inference formulation of [5] with a well-principled theoretical motivation (vs. the combination heuristics proposed in [5]). There is a consistent theoretical story behind the learning framework which is interesting (and I appreciate that all the proofs [except for theorem 5 -- see below] are provided. The paper is fairly clear to read and well-organized;  on the other hand, there are several technical assumptions made for which it unclear how crucial they are for their results. They also seem more geared towards the multilabel setup than a general structured prediction problem. For example, they only consider the 0-1 loss in the theory (and not Hamming e.g.); and they assume that the weight vectors are not shared across the edges in the graph (unlike is often the case in typical structured prediction applications). They also assume that the features are normalized on each edge. How important are these assumptions for their results? I would like the authors to comment on this in their rebuttal.I am not knowledgeable of the multilabel classification applications, so I cannot judge how significant are their empirical results. But the methodology seems sound; and it was interesting to observe that often the K-best lists were sufficient to obtain the global max.Pros:-A fairly complete theory motivating their learning objective.- Combinations of random spanning trees seem like a reasonable approach to avoid working with the full graph when the graph is unknown given their explanations.- Reasonable experimental comparison with somewhat practical algorithms.Cons:- Several technical assumptions are made along the way for which it is unclear how important they are to their results (or they are just there for simplicity).- The authors don't really relate their work to the existing literature apart saying that the resemblances are "superficial". == Other comments ==- Line 204: they cite [8] for the risk bound -- but in [8] (e.g. Theorem 4.17), the risk bounds are only for binary labels as far as I could find. How do the authors adapt it to the exponential number of multilabels? Can they provide a more specific pointer?=== Update after author feedback === Thank you for the clarifications.My comment about the technical assumptions was to clarify which conditions are for simplicity of presentation or simplifying the statements, and which ones are essential to the current proofs. Indeed, I was referring to the normalization assumption for the feature map. In particular, a different normalization could be used in practice (even though "it can be performed for most phi"); and so could a different normalization be easily handled? Similarly, weight sharing and non 0-1 loss are important in structured prediction applications; how hard would it be to extend the results to these settings? The authors should clarify this in their final version (as well as the other aspects that they said they would clarify).