Active Learning for Probabilistic Hypotheses Using the Maximum Gibbs Error Criterion
We introduce a new objective function for pool-based Bayesian active learning with probabilistic hypotheses. This objective function, called the policy Gibbs error, is the expected error rate of a random classifier drawn from the prior distribution on the examples adaptively selected by the active learning policy. Exact maximization of the policy Gibbs error is hard, so we propose a greedy strategy that maximizes the Gibbs error at each iteration, where the Gibbs error on an instance is the expected error of a random classifier selected from the posterior label distribution on that instance. We apply this maximum Gibbs error criterion to three active learning scenarios: non-adaptive, adaptive, and batch active learning. In each scenario, we prove that the criterion achieves near-maximal policy Gibbs error when constrained to a fixed budget. For practical implementations, we provide approximations to the maximum Gibbs error criterion for Bayesian conditional random fields and transductive Naive Bayes. Our experimental results on a named entity recognition task and a text classification task show that the maximum Gibbs error criterion is an effective active learning criterion for noisy models.
Overview: The authors propose the Gibbs error 
      criterion for active learning; seeking the samples that maximize the 
      expected Gibbs error under the current posterior. They propose a greedy 
      algorithm that maximises this criterion (Max-GEC). The objective reduces 
      to maximising a specific instance of the Tsallis entropy of the predictive 
      distribution which is very similar to Maximum Entropy Sampling (MES) which 
      uses the Shannon entropy of the predictive distribution. They consider the 
      non-adaptive, adaptive and batch settings separately, and in each setting 
      they prove using submodularity results that the greedy approach achieves 
      near-maximal performance compared to optimal policy. They show how to 
      implement their fully adaptive policy (approximately) in CRFs with 
      application to named entity recognition, and implement the batch algorithm 
      with a Naive Bayes classifier, with application to a text classification 
      task. Quality: Their algorithm appears to be a sensible 
      approach, with the Gibbs error being very closely related to the Bayes 
      error. Although it appears very similar to a number of approaches in the 
      literature (MES, Query by Committee), it exhibits some useful theoretical 
      and practical properties. Primarily, they are able to show near-optimal 
      bounds in the adaptive greedy setting with probabilistic predictions; 
      previous approaches have been shown only to be optimal in the non-adaptive 
      or noiseless case. A second advantage is that their objective permits a 
      simpler computation of their objective with CRFs when approximating 
      integrals over the posterior with samples; it would be interesting to see 
      a discussion of whether this rearrangement is extensible to other models? 
      However, I have a concern about the practicality of the algorithm. 
      In particular in the non-adaptive/batch case, a sum over the product space 
      of all possible labellings of the batch (S) is required (Eqns. 2, 4). When 
      applying batch Max-GEC to the NB classifier they approximate the sum using 
      Gibbs sampling. Given how large the space of possible labelings of the 
      batch could be, this may require a very large number of samples to get a 
      reasonable estimate, the requirement to compute/estimate this sum seems to 
      restrict the batch algorithm to either small batches or models in which 
      computing predictions is cheap. The experiment section is fairly 
      convincing; using two tasks and a number of different datasets, they show 
      max-GEC usually outperforms a number of baselines, although the 
      improvement over LeastConf seems marginal at best. A concern I have is 
      that for the CRF model, SegEnt/LeastConf produce almost as good results as 
      Max-GEC, which is perhaps unsurprising given the similarity between the 
      algorithms, however, for SegEnt/LeastConf the authors use only the MAP 
      hypothesis to compute uncertainties, and for Max-GEC they show improvement 
      by integrating over parameters (using samples). They should compare also 
      to SegEnt and LeastConf with parameter integration, these approaches may 
      be more sensitive to accurate estimation of uncertainties and I am 
      unconvinced that there would necessarily be a performance difference after 
      accounting for parameter uncertainty. Clarity: The paper is 
      largely clearly written, however, although they do define the notation, I 
      find some of the choices of notation somewhat confusing. In particular use 
      of a generic p to denote the posterior as opposed to the prior p_0 is a 
      bit unclear (perhaps it could be subscripted by the data used to train the 
      model). Also the use of the symbol traditionally used for ‘conditioning’ 
      (as in y_{A|X}) to denote the labelling of X according to A makes the 
      paper harder to read. The use of E(\rho) to denote the set of unlabelled 
      examples looks a lot like an expectation, also regarding expectations it 
      would be useful to subscript them with the distribution over which the 
      expectation is being taken e.g. for an expectation under the prior E_y 
      -> E_{p_0(y)}. Significance: Although the optimality proofs 
      provide an interesting insight into this particular criterion, and provide 
      a decent theoretical contribution, the algorithm itself is sufficiently 
      similar to a number of proposed approaches and so this paper does not 
      represent a very significant practical contribution to the field of active 
      learning.