Pruning Random Forests for Prediction on a Budget
We propose to prune a random forest (RF) for resource-constrained prediction. We first construct a RF and then prune it to optimize expected feature cost & accuracy. We pose pruning RFs as a novel 0-1 integer program with linear constraints that encourages feature re-use. We establish total unimodularity of the constraint set to prove that the corresponding LP relaxation solves the original integer program. We then exploit connections to combinatorial optimization and develop an efficient primal-dual algorithm, scalable to large datasets. In contrast to our bottom-up approach, which benefits from good RF initialization, conventional methods are top-down acquiring features based on their utility value and is generally intractable, requiring heuristics. Empirically, our pruning algorithm outperforms existing state-of-the-art resource-constrained algorithms.
This paper proposes a method for pruning a forest of trees taking into account feature costs. Pruning is formulated as a 0-1 integer program with linear constraints and it is shown that this program can be solved exactly through LP relaxation. Experiments are carried out on several datasets. 
The idea of taking into account feature costs when pruning tree ensembles is original to the best of my knowledge. The main originality of the proposed approach is the fact that it adopts a bottom-up post-pruning strategy, while most existing approaches are top-down, acting during tree growing. While the authors present this feature as an advantage of their method, actually, I'm not convinced that adopting a bottom-up strategy is a good idea for addressing this problem. Since the algorithm indeed can not modify the existing tree structure (it can only prune it), it should be less efficient in terms of feature cost reduction than top-down methods that can have a direct impact on the features selected at tree nodes. For example, let us assume that two very important features in the dataset carry on the exact same information about the output (i.e, they are redundant). They can both be included in a forest if this latter is built without constraint and BudgetPrune will have no choice but to keep these two variables in the model, even though only one of them is actually necessary.    The optimization problem is nicely formulated, in particular in its coupling of the feature usage variables. The proposed primal-dual optimization problem seems sound and very efficient as it scales linearly with respect to the number of trees (compared to the initial cubic LP relaxation). In the formulation of the (IP) problem, I'm surprised by the fact that the error term in the objective function is the average of the individual errors of the ensemble trees and not the error of the ensemble itself. I guess this is the main reason why the optimization problem can be solved efficiently by the primal-dual algorithm but I think that the authors should discuss and motivate this choice better in the paper. Another  not so well-motivated design choice is the incorporation of the computational cost (average leaf depths) in the objective function. Given that the experiments only focus on assessing the average feature cost, is it necessary to include the computational cost? Doesn't it harm the average feature cost? Don't you need also to introduce an additional coefficient in the objective function to balance the feature acquisition costs and computational cost? Merely summing them seems risky without a proper normalization.    Experiments seem to have been well conducted. They are extensive enough, although the number of datasets (4) could have been higher. They rather convincingly show the superiority of BudgetPrune over competitors (in Figure 2). Experiments in the supplementary material are also very useful to complement the experiments in the main paper.     Some points however need clarification before the results can really be assessed: - The authors propose to use in-bag data to estimate error probability at each node and the validation data for the feature cost variables. It's not clear to me what it means. What is the in-bag data? Is it the learning sample itself or the bootstrap sample of each tree? - It's not clear if all methods make use of the validation data. For the methods that do not need a validation set, it would be more fair to train them on the whole training+validation data. - In the CCP method, what do the authors mean by the sentence "we perform CCP on individual trees to different levels to obtain various points on the accuracy-cost tradeoff curve"? This should be more precisely explained. - The GreedyPrune method is also unclear. What does it mean to remove all nodes corresponding to one feature? Do you prune the whole subtree below each split on this feature? If not, what do you do with the left-right successor nodes of this split? What does it mean to select the pruned RF that "has the lowest training error and average feature cost"? Do you sum these two scores? With a weight? - In the plots in Figure 2,how is the average feature cost measured on the x-axis? Also, how are different tradeoffs obtained for the different methods in Figure 2? With BudgetPrune, I guess that the parameter lambda is changed to obtain a curve but how is it done for the other methods? Is there only one parameter that can change feature cost? - All methods are evaluated only in terms of average feature cost and accuracy while the objective function also includes a term related to the computational costs (the average depth of the leaves reached by the examples). Why are the approaches not evaluated also according to this criterion? Is this term really crucial in the objective function?   The paper is well-written and mostly clear (except for the missing details in the experiments). However, the fact that several experimental details are only present in the supplementary material makes reading section 5 not so easy. I think that some more details could have been given in the main paper.    Minor comments: - There is no reference in the main text to the fact that the supplementary material contain additional (and necessary) details about the problems and settings of algorithms. A link should be given at the beginning of Section 5. - Page 1, I don't understand the meaning of footnote 2. - In the case of uniform cost, alternative competitors to BudgetPrune are feature selection methods that tries to find a small subset of features that are sufficient to predict the output. It would interesting to include feature selection methods in the comparison (for example using Random forests variable importances). - In Section 5.1: "This is a fundamentally combinatorial problem...and we are able to globally optimize cost-accuracy objective". "our proposed formulation possesses 1) elegant theoretical properties".  The BudgetPrune algorithm indeed finds a global optimum but it needs to start from an existing random forests whose training is also a combinatorial problem solved in practice by using relaxations and heuristic with no guarantees of performance in the end. So, I'm not sure it's fair to say that BudgetPrune provides a better solution with stronger theoretical guarantees to the problem than top-down methods.