Latent Structured Active Learning
In this paper we present active learning algorithms in the context of structured prediction problems. To reduce the amount of labeling necessary to learn good models, our algorithms operate with weakly labeled data and we query additional examples based on entropies of local marginals, which are a good surrogate for uncertainty. We demonstrate the effectiveness of our approach in the task of 3D layout prediction from single images, and show that good models are learned when labeling only a handful of random variables. In particular, the same performance as using the full training set can be obtained while only labeling ∼10% of the random variables.
The authors extend the work of [Schwing et al., 
      Efficient Structured Prediction with Latent Variables for General 
      Graphical Models; ICML 2012] to include active learning protocols. In 
      essence, they use the entropy of the local variable marginals to estimate 
      a value of uncertainty -- so this can be viewed as local variable 
      uncertainty sampling for structured predictions with partially observed 
      variables. To this end, they propose two active learning variants, namely 
      (1) separate active: each active learning round requires a "separate" 
      inference step over the unlabeled and partially labeled variables after 
      learning on the observed variables and (2) joint active: each active 
      learning round follows a joint learning procedure over the labeled, 
      partially labeled, and unlabeled instances. They also explore batch sizes 
      and warm-starting of the learning procedure between rounds in this 
      setting. The empirical evaluation is 3D layout of rooms (a computer vision 
      problem explored in [Scwing et al., ICML 2012], demonstrating that (1) 
      "joint" active learning works the best, achieving an annotation savings of 
      ~90%, (2) batch mode works for reasonable small batches, (3) querying 
      "full" vs. "partial" labels [unsurprisingly, partial labels works better] 
      (4) sensitivity to \epsilon and (5) computational reuse saves time 
      [unsuprisingly]. Overall, it is a nice paper that shows active 
      learning works in a challenging setting with this particular formulation. 
      I will address the {quality, clarity, originality, significance} 
      components separately below. Quality: The paper is certainly of 
      high quality. The model for structured learning with latent variables is 
      state of the art and the active learning method achieves dramatic savings 
      over just using random sampling (which actually makes quite a bit of sense 
      in the latent variable case when you think about it). My biggest complaint 
      from an active learning perspective is that the saving are somewhat 
      disingenuous (even if not intentionally so) -- while the "$1 for 1 label" 
      simulations were very popular from ~1995-2008ish, I think it has been 
      accepted that these sorts of simulations aren't very realistic since 
      active learning often picks the most difficult examples for labeling 
      (e.g., [Kapoor et al., Selective supervision: Guiding supervised learning 
      with decision-theoretic active learning; IJCAI 2007],[Settles et al., 
      Active learning with real annotation costs; NIPS WS 2008], [Ringger et 
      al., Assessing the costs of machine-assisted corpus annotation through a 
      user study; LREC 2008], [Haertel et al., Return on investment for active 
      learning; NIPS WS 2008], [Arora et al., Estimating annotation cost for 
      active learning in a multi-annotator environment; NAACL WS 2009], 
      [Baldridge and Palmer, How well does active learning actually work? 
      Time-based evaluation of cost-reduction strategies for language 
      documentation; EMNLP 2009], [Wallace et al., Modeling Annotation Time to 
      Reduce Workload in Comparative Effectiveness Reviews; IHI 2010]...amongst 
      others. I think there should at least be some discussion along these lines 
      (although the full vs. partial labels experiments do at least hint at this 
      a bit). My second "active learning" concern is a bit on the semantics 
      side, but both [Culotta and McCallum, AAAI 2005] and [Roth and Small, ECML 
      2006] consider partial labels during querying time -- which is somewhat in 
      contrast to the last paragraph of page 1. I think what you mean to say is 
      that the output space is fully observable or something along these lines, 
      as they certainly do not require "full" labels from the querying function 
      (and I actually don't think this is what you meant) Clarity: For 
      the most part, the paper is clear -- although I had to go read [Scwing et 
      al., ICML 2012] to understand some of the details. The biggest thing I 
      noticed in this case is (1) I didn't completely understand the "task-loss" 
      function until I read the previous work; some example/discussion here 
      would be nice (2) I think the notation for this loss function may have 
      suffered from some cutting/pasting from the ICML paper as there appears to 
      be some confusion between s and y here and there (assuming I am 
      understanding everything correctly) and (3) you never say anything about 
      $\ell^c$ -- which may be difficult to digest for those without explicit 
      graphical models expertise. Originality: While the step is 
      straightforward (and based on uncertainty sampling), it hasn't been done 
      before in this setting. However, I also think anybody with knowledge of 
      these areas could have put these elements together. Significance: 
      I would guess that this paper will not have the same sort of impact as 
      [Scwing et al., ICML 2012] as it is almost a "companion piece". The ideas 
      are relatively straightforward, but done well. Probably having the code 
      would be more useful than reading the paper as there are no theoretical 
      results or anything along these lines and the setting of $\epsilon$ seems 
      non-trivial in deployed settings. Therefore, I would say it is 
      interesting, but not groundbreaking in this regard. A few small 
      comments: abstract: -> $\sim 10\%$ (get rid of the weird spacing) 
      pg 1: "have to deal with exponentially sized output variables" -> 
      "have to deal with output spaces which are exponentially sized in the 
      number of local variables" (or something along these lines) pg 2: 
      "that employ both fully labeled training sets" As I previously stated, I 
      think there is a subtle distinction between "labeled" and "observed" -- 
      but this is obviously your call if you agree pg 5: "as well as the 
      messages $\lambda$ and believes" -> "... beliefs"