Dual Decomposed Learning with Factorwise Oracles for Structural SVMs of Large Output Domain
Many applications of machine learning involve structured outputs with large domains, where learning of a structured predictor is prohibitive due to repetitive calls to an expensive inference oracle. In this work, we show that by decomposing training of a Structural Support Vector Machine (SVM) into a series of multiclass SVM problems connected through messages, one can replace an expensive structured oracle with Factorwise Maximization Oracles (FMOs) that allow efficient implementation of complexity sublinear to the factor domain. A Greedy Direction Method of Multiplier (GDMM) algorithm is then proposed to exploit the sparsity of messages while guarantees convergence to sub-optimality after O(log(1/ )) passes of FMOs over every factor. We conduct experiments on chain-structured and fully-connected problems of large output domains, where the proposed approach is orders-of-magnitude faster than current state-of-the-art algorithms for training Structural SVMs.
This paper introduces an approach to learning of structural SVMs based on a direction method of multipliers. The starting point is to bring the primal form of the learning objective into a dual-decomposed representation (eq. 9), based on the LP relaxation of the inference problem. This representation is frequently used for inference in intractable graphical models, and has meanwhile also been exploited successfully for structured learning (e.g. in [13,14]). Basically, the problem is broken down into maximization of individual factors, as well as minimization with respect to the model parameters as well as Lagrange variables coupling the factors.  Approaches then differ in how the coupling constraint are handled, and in how the minimization with respect to the model parameters is conducted.  In the present work, the objective in (eq. 9) is again dualized, yielding an objective that is basically a sum of dual SVM objectives - which needs to optimized over variables subject to simplex constraints (as usual in the dual SVM representation), as well as additional coupling constraints resulting from the individual SVMs also being coupled in the primal representation.  The coupling constraints are again handled by forming the Lagrangian; and updates with respect to the Lagrange multipliers are then performed by gradient descent. For minimizing the augmented Lagrangian function, the authors propose two different strategies: 1) Fully-corrective block-coordinate Frank-Wolfe (for factors/sub-SVMs of large cardinality) 2) Block-greedy coordinate descent (for a large number of factors/sub-SVMs)  For factors of large cardinality, it is sometimes possible to find the maximizing configuration in sub-linear time. The authors give a few examples of where this is the case. However, these are not novel insights to my knowledge but have been exploited before. (See, e.g. Martins et al.: "AD3: Alternating Directions Dual Decomposition for MAP Inference in Graphical Models.", JMLR 16 (2015) 495-545). This maximization in sub-linear time can be exploited in the suggested learning algorithms.  The authors perform an empirical comparison against several recent approaches, which suggest that their approach is highly competitive.  
This manuscript definitely introduces some new ideas, and the empirical performance of the proposed algorithms seem strong. A lot of work was also put into the convergence proofs. These are non-trivial technical contributions that I find very valuable.  However, I feel that the presentation lacks clarity and polish, and that there is a lack of rigor in the experimental evaluation:  1) There is no conclusion. It is not clear to me why the authors omitted it, as it seems there would have been some space for this still. The conclusion is very helpful for the reader in order to put things into perspective, so I strongly suggest to include one.  2) The description of the experiments is very brief and lacks important information. For instance, it is not clear to me how the \rho parameter of Soft-BCFW was chosen, or how the other competing algorithms were parameterized. I also do not see any mention of the step size parameter \gamma required by the authors' own algorithm. There needs to be a clean protocol for choosing hyper parameters.  3) I am not certain that the experimental comparison was fair. For the comparison in terms of computation time, the authors only plot their "GDMM-subFMO" approach, which exploits sub-linear maximization complexity of certain factor types. The competing algorithms, on the other hand, do not seem to exploit this, even though in my understanding, Soft-BCFW also only requires a factor maximization oracle, and hence could be made to benefit from exactly the same strategy. To better put things into perspective, I would like to see a comparison of "naive" GDMM (without subFMO) with the other algorithms, in terms of computation time.  The claim of the authors is quite strong ("The proposed approach is orders-of-magnitude faster than the state-of-the-art training algorithms for Structural SVM"), so I think there must be a bullet-proof experimental validation that can also be reproduced by others. In the current state of the manuscript, I do not think that this is the case.