Auditing: Active Learning with Outcome-Dependent Query Costs
We propose a learning setting in which unlabeled data is free, and the cost of a label depends on its value, which is not known in advance. We study binary classification in an extreme case, where the algorithm only pays for negative labels. Our motivation are applications such as fraud detection, in which investigating an honest transaction should be avoided if possible. We term the setting auditing, and consider the auditing complexity of an algorithm: the number of negative labels the algorithm requires in order to learn a hypothesis with low relative error. We design auditing algorithms for simple hypothesis classes (thresholds and rectangles), and show that with these algorithms, the auditing complexity can be significantly lower than the active label complexity. We also show a general competitive approach for learning with outcome-dependent costs.
Overview: The authors address the problem of 
      auditing, a restriction of the more general class-dependent cost sensitive 
      active learning for binary classification where one pays for receiving 
      labels on just the ‘negative’ examples. The paper presents a number of 
      theoretical results for ‘auditing complexity’ (the number of negative 
      labels required). Firstly, they note that, like active learning, in the 
      general case auditing cannot necessarily improve over passive learning. 
      Their main result is to show improved auditing complexity over active 
      label complexity for threshold and axis aligned rectangular decision 
      boundaries (both in the realisable and noisy, agnostic case). Finally they 
      present a general algorithm to adapt a greedy active learner to auditing 
      achieving constant speedup and the same complexity as the learner of 
      Golovin, Krause (2011). Quality: In this work they present new 
      complexity results, and the algorithms by which they may be achieved. The 
      theoretical content of the paper is thorough and the algorithms presented 
      yield insight into the differences between the auditing and active 
      learning problem. The main results, and algorithms provided, apply only to 
      highly restricted settings that are unlikely to be of much use in 
      practice. They show improved complexity of auditing only for data that can 
      be classified with axis aligned thresholds, or rectangles - with a further 
      restriction that the algorithm only works if the negative examples are on 
      the inside of the rectangle. In the noisy case (as discussed in the paper) 
      the algorithm must be provided with the noise level in advance. Therefore 
      although the theoretical results are insightful, these restrictions appear 
      to strong to be of practical relevance. One concern I have is with 
      the agnostic case for the rectangular classification boundary - although 
      there is an exponential reduction in complexity with respect to the noise 
      level \eta, there is a polynomial increase the dimension, d. It is not 
      clear to me in practice whether this trade-off will necessarily be 
      beneficial, further discussion of typical values (or even an empirical 
      evaluation) would make the case that auditing complexity is better than 
      the standard active sampling complexity more convincing. Although 
      the paper is theoretical in nature, I find the complete lack of 
      experimental evaluation of their proposed algorithms disappointing. Even 
      if only on toy data, if the authors could show empirically the improved 
      auditing performance of their auditing algorithms over both passive and 
      active sampling it would make the paper more impactful to the more general 
      reader. Clarity: The paper is clearly written and intuitions 
      are provided to assist the reader in understanding the high-level workings 
      of the algorithms and exploitable structure for auditing in the scenarios 
      addressed in the paper. Originality: Although auditing has 
      been indirectly addressed as it is a specific instance of cost-sensitive 
      active learning, addressing the problem directly seems novel and useful. 
      The application to fraud presented in the introduction is convincing, 
      perhaps the authors could provide a could more motivating examples here? 
      The authors claim that in existing work in cost-sensitive active 
      learning the costs need to be known in advance - however, for example, in 
      the work of Kapoor et al (2007) the cost may be class dependent which is 
      unknown at time of sampling, they take expectations of the cost function 
      to account for this uncertainty. Surely their method could be directly 
      applied to the auditing scenario too? Significance: The paper 
      provides new theory on auditing, which is a relatively unstudied area. 
      Although I feel that the algorithms provided have little practical 
      significance the results presented in this paper yield insight into the 
      auditing problem and could provoke further study into the practical side 
      of active auditing.