Learning Active Learning from Data
In this paper, we suggest a novel data-driven approach to active learning (AL). The key idea is to train a regressor that predicts the expected error reduction for a candidate sample in a particular learning state. By formulating the query selection procedure as a regression problem we are not restricted to working with existing AL heuristics; instead, we learn strategies based on experience from previous AL outcomes. We show that a strategy can be learnt either from simple synthetic 2D datasets or from a subset of domain-specific data. Our method yields strategies that work well on real data from a wide range of domains.
Key insights/novelty: 
* Trains a regression model that predicts improvement in generalisation error
* To the best of my knowledge this is an original approach to this problem (although similar, but sufficiently different, methods do exist)
Questions:
* In the experimental section, I recommend that the authors also include experiments on rotated checkerboards - the discriminating features of checkerboards are aligned to the axes, and using random forest classifiers may yield an overly optimistic performance curve. Rotating at 45 degrees may yield overly pessimistic curve, but it will explain the utility of the proposed methods better. 
* Baseline methods are not satisfactory since none of the proposed baselines can learn as much as the topology of the feature space as the proposed method (eg Zhu, Ghahramani, Lafferty ICML2003). At the very least this method should be compared in the discussions. 
* I'm not convinced by the results since they don't seem to be significantly better than baselines. Particularly with uncertainty sampling on the real data. This point relates to my suggestion of rotating the checkerboard so that the discriminating features will not be axis-aligned. 
* This method will require a relatively substantial proportion of data to be labelled to begin with. While you say that only a couple of hours were spent in labeling for the offline experiments, this time is domain-specific. Are there any techniques that can be used to improve this? 
* The conclusions of this paper were very short and underwhelming. 
General Statements: 
* Very well written, clear motivation. Easy to follow story. 
Small typos/recommended corrections: 
* L79. Unannotated data? 
* Showing the relative accuracy only shows one side of the picture. It would be insightful to show the accuracies of both too. Practitioners will be interested in knowing whether it is normally with learning strategies that have a good initial generalisation that disimprove moving forward. 
* Algorithm 1: unclear which datasets the classifiers are trained on. Recommend clarifying. And in line 14, please clarify. Do you mean $\{\delta_m: 1 \leq m \leq M\}$? 
* Should a selection function be passed into algorithm 1 to clarify line 7?