Adaptive Classification for Prediction Under a Budget
We propose a novel adaptive approximation approach for test-time resourceconstrained prediction motivated by Mobile, IoT, health, security and other applications, where constraints in the form of computation, communication, latency and feature acquisition costs arise. We learn an adaptive low-cost system by training a gating and prediction model that limits utilization of a high-cost model to hard input instances and gates easy-to-handle input instances to a low-cost model. Our method is based on adaptively approximating the high-cost model in regions where low-cost models suffice for making highly accurate predictions. We pose an empirical loss minimization problem with cost constraints to jointly train gating and prediction models. On a number of benchmark datasets our method outperforms state-of-the-art achieving higher accuracy for the same cost.
1. Summary of paper
This paper introduced an adaptive method for learning two functions. First, a `cheap' classifier that incurs a small resource cost by using a small set of features. Second, a gating function that decides whether an input is easy enough to be sent to the cheap classifier or is too difficult and must be sent to a complex, resource-intensive classifier. In effect, this method bears a lot of similarity to a 2-stage cascade, where the first stage and the gating function are learned jointly.
2. High level subjective
The paper is written clearly for a reader familiar with budgeted learning, although I think a few parts of the paper are confusing (I'll go into this more below). I believe the idea of approximating an expensive, high-accuracy classifier is new (cascades are the most similar, but I believe cascades usually just consider using the same sort of classifiers in every stage). That said, it is very similar to a 2-stage cascade with different classifiers. I think there are certainly sufficient experiments demonstrating this method. Likely this method would be interesting to industry practitioners due to the increased generalization accuracy per cost.
3. High level technical
Why not try to learn a cost-constrained q function directly? If you removed the KL term and used q in the feature cost function instead of g, wouldn't you learn a q that tried as best to be accurate given the cost constraints? Did you learn a g separately in order to avoid some extra non-convexity? Or is it otherwise crucial? I would recommend explaining this choice more clearly.
I would have loved to see this model look less like a cascade. For instance, if there were multiple expensive 'experts' that were learned to be diverse w.r.t each other and then multiple gating functions along with a single cheap classifier were learned jointly, to send hard instances to one of the experts, then this would be a noticeable departure from a cascade. Right now I'm torn between liking this method and thinking it's a special cascade (using techniques from (Chen et al., 2012) and (Xu et al., 2012)).
4. Low level technical
- Line 161/162: I don't buy this quick sentence. I think this is much more complicated. Sure we can constrain F and G to have really simple learners but this won't constrain computation per instance, and the only guarantee you'll have is that your classifier is at most as computationally expensive as the most expensive classifier in F or G. You can do better by introducing computational cost terms into your objective. 
- Line 177: There must be certain conditions required for alternating minimization to converge, maybe convexity and a decaying learning rate? I would just clarify that your problem satisfies these conditions.
- I think Figure 5 is a bit confusing. I would move Adapt_Gbrt RBF to the top of the second column in the legend, and put GreedyMiser in its original place. This way for every row except the last we are comparing the same adaptive/baseline method.
5. Summary of review
While this method achieves state-of-the-art accuracy in budgeted learning, it shares a lot of similarities with previous work (Chen et al., 2012) and (Xu et al., 2012) that ultimately harm its novelty.