Search Improves Label for Active Learning
We investigate active learning with access to two distinct oracles: LABEL (which is standard) and SEARCH (which is not). The SEARCH oracle models the situation where a human searches a database to seed or counterexample an existing solution. SEARCH is stronger than LABEL while being natural to implement in many situations. We show that an algorithm using both oracles can provide exponentially large problem-dependent improvements over LABEL alone.
Counterexamples can be ineffective or misleading in practice as well. Reconsidering the intervals example above, a counterexample on the boundary of an incorrect interval provides no useful information about any other examples. What is a good counterexample? What is a natural way to restrict the searchable space? How can the intervals problem be generalized?
We define a new oracle, SEARCH, that provides counterexamples to version spaces. Given a set of possible classifiers H mapping unlabeled examples to labels, a version space V ⊆ H is the subset of classifiers still under consideration by the algorithm. A counterexample to a version space is a labeled example which every classifier in the version space classifies incorrectly. When there is no counterexample to the version space, SEARCH returns nothing.
How can a counterexample to the version space be used? We consider a nested sequence of hypothesis classes of increasing complexity, akin to Structural Risk Minimization (SRM) in passive learning [see, e.g., Vapnik, 1982, Devroye et al., 1996. When SEARCH produces a counterexample to the version space, it gives a proof that the current hypothesis class is too simplistic to solve the problem effectively. We show that this guided increase in hypothesis complexity results in a radically lower LABEL complexity than directly learning on the complex space. Sample complexity bounds for model selection in LABEL-only active learning were studied by Balcan et al. [2010], Hanneke [2011]. SEARCH can easily model the practice of seeding discussed earlier. If the first hypothesis class has just the constant always-negative classifier h(x) = −1, a seed example with label +1 is a counterexample to the version space. Our most basic algorithm uses SEARCH just once before using LABEL, but it is clear from inspection that multiple seeds are not harmful, and they may be helpful if they provide the proof required to operate with an appropriately complex hypothesis class.
Defining SEARCH with respect to a version space rather than a single classifier allows us to formalize "counterexample far from the boundary" in a general fashion which is compatible with the way LABEL-based active learning algorithms work.
Related work. The closest oracle considered in the literature is the Class Conditional Query (CCQ) [Balcan and Hanneke, 2012] oracle. A query to CCQ specifies a finite set of unlabeled examples and a label while returning an example in the subset with the specified label, if one exists.
In contrast, SEARCH has an implicit query set that is an entire region of the input space rather than a finite set. Simple searches over this large implicit domain can more plausibly discover relevant counterexamples: When building a detector for penguins in images, the input to CCQ might be a set of images and the label "penguin". Even if we are very lucky and the set happens to contain a penguin image, a search amongst image tags may fail to find it in the subset because it is not tagged appropriately. SEARCH is more likely to discover counterexamples-surely there are many images correctly tagged as having penguins.
Why is it natural to define a query region implicitly via a version space? There is a practical reason-it is a concise description of a natural region with an efficiently implementable membership filter [Beygelzimer et al., 2010, 2011, Huang et al., 2015. (Compare this to an oracle call that has to explicitly enumerate a large set of examples. The algorithm of Balcan and Hanneke [2012] uses samples of size roughly dν/�
This article studies a type of active learning in which the algorithm has access both to the usual Label query oracle and also a "Search" oracle.  The Search oracle takes a set of classifiers as input, and returns a correctly labeled example that contradicts all of them, if one exists.  They first relate this oracle to others that have appeared in the literature previously, and provide several nice examples.  They then propose a general active learning algorithm that uses a nested sequence of hypothesis classes, and uses the Search oracle to perform model selection, and to construct a seed data set for an active learning algorithm for each hypothesis class.  They study this both in the realizable case and the agnostic case.
Putting aside the practicality of this framework, the theoretical analysis of the advantages of access to this type of oracle is interesting.  This work reveals that the search oracle provides substantial advantages in the context of model selection.  For instance, by making roughly k* search queries (where k* is the index of the model containing the Bayes classifier), Theorem 3 reveals that the number of label queries can be reduced by at least a factor of k* (and sometimes dramatically more) compared to the best known result in the prior literature on model selection in active learning (where only label queries are allowed).  This can be a significant reduction in some settings.  Now, on the issue of practicality, the authors make an effort to motivate each major component of the framework in terms of practical considerations.  However, in many cases, these motivations seem fairly thin.  A good example is the fact that, even in the agnostic case, the search oracle returns the label h*(X) of the optimal classifier h* at the returned point X, rather than the (noisy) label Y associated with that example X.  The motivation given is that we can suppose the oracle will choose an example for which she is supremely confident in the label.  But this motivation implicitly assumes that the only source of noise is the annotator's uncertainty about the label; this is often not the case, as factors such as the feature space representation can also introduce inherent noise (e.g., in NLP, choosing a bag-of-words representation creates noise, since the human annotator sees the words in order, and different word orders may form different valid sentences, and which of these orders is observed may affect her choice of label, so that the conditional distribution of label given the unordered list of words may have nonzero entropy). That said, I can appreciate that it is sometimes desirable for the  annotator to be able to indicate a strong preference for a particular  label, and supposing this strongly-preferred label to be h*(X) might  be a reasonably elegant way to incorporate this extra information  into the formal model, and that in some cases the h*(X) assumption  might not need to strictly hold for the algorithm to have reasonable  behavior in practice.  Overall, I found this to be an interesting piece of work, definitely worth reading for those interested in the theory of interactive learning.  However, I would expect any future works that build on it will probably tweak various aspects of the model, to weaken the  search oracle, or potentially modify the type of input (or the restrictions thereon) to the oracle (e.g., I can imagine these search constraints, based on the region of agreement of a subset of H, could potentially make a minimax lower bound analysis significantly more difficult than it would be for certain natural alternatives, some  of which make more practical sense and still allow for the types of regions used in the algorithms proposed here).   minor comments:  In the related work section, the notation d \nu/\epsilon^2 is used without being defined or explained.  The main theorems in this paper are somewhat difficult to interpret in the abstract, being sample-dependent bounds on the number of labeled samples the algorithm uses.  Is there any hope of obtaining concise abstract a-priori bounds, as were possible for the unions of intervals example?  The wording in the discussion section is confusing to me.  It seems to me that the search oracle would often be more powerful (provides  more information) than the CCQ oracle, since CCQ only needs to return  the noisy label while search returns the h* label.  I guess the discussion is referring to the fact that CCQ can specify any region, not only a region of agreement of a subset of H (and in particular, in the realizable case this means CCQ is at least as powerful as search).  So discussing these two types of queries as being a kind of trade-off in the agnostic case seems more accurate than what is currently written.  Related to this, it might be interesting to discuss teasing apart the contribution of getting h*(X) labels instead of Y labels, vs the inherent power in these kinds of region-constrained mistake queries (i.e., with Y labels returned).  Can one show lower bounds for region-constrained mistake queries where the Y labels are returned instead of h* labels, worse than the upper bounds proven here using the search oracle?  The bibliography is inconsistent in its choice of whether or not to abbreviate author first names.