Structured Learning via Logistic Regression
A successful approach to structured learning is to write the learning objective as a joint function of linear parameters and inference messages, and iterate between updates to each. This paper observes that if the inference problem is "smoothed" through the addition of entropy terms, for fixed messages, the learning objective reduces to a traditional (non-structured) logistic regression problem with respect to parameters. In these logistic regression problems, each training example has a bias term determined by the current set of messages. Based on this insight, the structured energy function can be extended from linear factors to any function class where an "oracle" exists to minimize a logistic loss.
The authors address the issue of learning unary and 
      pairwise parameters for discrete valued random fields in a structured 
      output SVM setting. In order to enable learning with non-linear function 
      classes, the authors remove the standard l2 regularization and replace it 
      with an entropy regularization that decomposes over the potentials of the 
      random field. This formulation results in a saddle-point problem (typical 
      for structured prediction) in which parameters of an LP relaxation are 
      optimized along with parameters of the (non-linear) discriminant function. 
      The authors use a message passing formulation, and show that with their 
      decomposable loss, and decomposable regularizer, the problem decomposes 
      into problems that each resemble logistic regression. Any solver for a 
      non-linear function class can now be applied. Experimental results are 
      given on relatively simple denoising and horse segmentation datasets 
      showing that potentials modeled with a multi-layer perceptron performed 
      better than other linear and non-linear methods. Quality: The 
      derivation is principled and complete, with sufficient references to the 
      optimization literature. The idea is interesting, and (given a source code 
      release) could result in various applications of non-linear function 
      classes in learning the parameters of graphical models. The quality of the 
      experimental validation is somewhat weak, however, given that the datasets 
      are quite simple, and results only compare with variants of the proposed 
      algorithm. Plenty of citations are given for competing methods, e.g. [18] 
      which is very closely related to the proposed approach in its goal of 
      enabling non-linear function classes for random field models. 
      Clarity: The article is well written, and clear throughout. 
      Originality: [18] have previously proposed a random forest 
      framework for learning the parameters of a random field model. This work 
      is more general and uses a different formulation to arrive at its 
      objective function. Significance: This work is quite general 
      in the functions that can be used, though a number of limiting assumptions 
      are made (decomposable loss, margin rescaling, only consider discrete 
      labelings). It is an interesting approach nevertheless. Additional 
      comments: Sec 3 - Limiting assumptions about the problem setting are 
      first introduced here: Eq (3) - only considers margin rescaling 
      Around line 93 - \Delta decomposes over \alpha - e.g. Hamming distance 
      Around line 94 - only considering discrete labelings - which is fine 
      but not fully general in structured prediction Decomposable loss and 
      discrete labelings should probably already be mentioned in the 
      introduction Line 306 - typo in equation