Embedding Inference for Structured Multilabel Prediction
A key bottleneck in structured output prediction is the need for inference during training and testing, usually requiring some form of dynamic programming. Rather than using approximate inference or tailoring a specialized inference method for a particular structure-standard responses to the scaling challengewe propose to embed prediction constraints directly into the learned representation. By eliminating the need for explicit inference a more scalable approach to structured output prediction can be achieved, particularly at test time. We demonstrate the idea for multi-label prediction under subsumption and mutual exclusion constraints, where a relationship to maximum margin structured output prediction can be established. Experiments demonstrate that the benefits of structured output training can still be realized even after inference has been eliminated.
This paper deals with a particular task of structured prediction: binary variables, score function consisting of unary potentials only, structure consists in the feasibility constraints (logical and mutual exclusion). The paper attempts to get rid of the costly inference at the test and train stages by posing special constraints on the model parameters such that trivial inference procedure automatically satisfies the feasibility constraints. The proposed approach is compatible with multiple structured losses, e.g. conditional log-likelihood (2) and structured large margin (3) are mentioned. To the best of my knowledge the idea is original and seems interesting. The paper is relatively clear although sometimes it is hard to follow the technical details (I have several major comments/questions about clarity/correctness, see below). The idea of embedding inference might be quite influential (lead to structured prediction methods without inference) even given that the direct applications of the paper are quite narrow.Major: 1)Results of sections 3.2 and 3.3 (key results of the paper) are based on Lemma 2, which has no proof. Lemma 2 consists of claims (9) and (10). I agree that (9) is analogous to lemma 1 and therefore clear. Equation (10), however, appears for the first time and in my opinion seems non-trivial. If it does not hold (under the assumptions of lemma 2) than section 3.2-3.3 are incorrect.2) One of the contributions of the paper (mentioned in the abstract) is "equivalence to maximum margin structured output prediction". I do not see how this follows from the results of the paper. In particular, proposition 6 claims that is the score function satisfies constraints (5) and (6) than the large-margin objective can be computed without performing inference (r.h.s. of (22)). But general claim of equivalence in my opinion should sound something like this: "A minimizer of objective (3) (can be simplified by (22) under all the assumptions) under model (8) with theta_PAQ being positive semi-definite and constraints (11), (12), (16) (imposing (5) and (6)) at the same time minimizes objective (3) under model (8) with theta_PAQ being positive semi-definite (constraints are satisfied by inference)". I'd really like the authors to clarify what is meant by "equivalence". 3) In the experiments it is not specified how the hyper-parameters of the methods were chosen. In particular, choice of the weight of the regularizer r(theta) could significantly influence the results. Minor: Line 153. I do not understand why s_k(x) \geq 0 is a substantial requirement. Lines 159 and 319. Notation "argmax_{k \in {0,1} } f(k)= 1" is a bit confusing i.e. it is not clear what happens if f(1) = f(0). I suggest writing "1 \in argmax_{k \in {0,1} }". Line 295. Notation "t_{ik}" (two indices) is not introduced. Being slightly more careful would help the reader.*Comments after the response Thanks for for prooving Lemma 2 and clarifying Proposition 6.Please, incorporate these comments into the paper. In particular, phrase from the abstract "We demonstrate this idea for multi-label prediction under subsumption and mutual exclusion constraints, where equivalence to maximum margin structured output prediction is established." sounds like an overclaim.