Efficient Non-greedy Optimization of Decision Trees
Decision trees and randomized forests are widely used in computer vision and machine learning. Standard algorithms for decision tree induction optimize the split functions one node at a time according to some splitting criteria. This greedy procedure often leads to suboptimal trees. In this paper, we present an algorithm for optimizing the split functions at all levels of the tree jointly with the leaf parameters, based on a global objective. We show that the problem of finding optimal linear-combination (oblique) splits for decision trees is related to structured prediction with latent variables, and we formulate a convex-concave upper bound on the tree's empirical loss. Computing the gradient of the proposed surrogate objective with respect to each training exemplar is O(d 2 ), where d is the tree depth, and thus training deep trees is feasible. The use of stochastic gradient descent for optimization enables effective training with large datasets. Experiments on several classification benchmarks demonstrate that the resulting non-greedy decision trees outperform greedy decision tree baselines.
The submission proposes an algorithm for global optimization of decision trees based on a reformulation of the problem as (latent) structured prediction. The latent structured variables are sign vectors encoding the decisions made at each internal node in the tree.A non-differentiable and non-convex loss is proposed, which is then approximated by a differentiable (but still non-convex) upper bound.Minimizing this loss via SGD entails solving a sequence of loss-augmented inference problems, for which efficient algorithms are given. Experiments are shown comparing this method to a standard greedy training method.The key contributions of the work are casting the global DT optimization as structured prediction and devising efficient algorithms to optimize the objective.The way this is done is novel and interesting, and raises interesting questions.I have some concerns, however, about some practical considerations-especially concerning the motivation for the method and how significant the performance gains over naive methods are.Considered purely as a method to globally optimize DTs, the method is intriguing in several ways.The first advantage gained by casting the optimization as structured prediction is being able to perform SGD by solving a sequence of loss-augmented optimization problems that leverage the special structure of the problem.However, I think the most interesting result of this approach is that the gradient updates are sparse-if I'm not mistaken, each gradient step involves changing only one node's weight vector.The submission presents this mainly as a computational advantage, but I believe that this property raises more intriguing possibilities that are not really acknowledged in the paper.One could imagine optimizing over an infinitely deep tree by initially setting all weights to zero, and then gradually growing the tree by making more weights nonzero, thus letting the model's complexity grow naturally to fit the data.This raises interesting questions: for example, would such a method prove to be equivalent to, or a variation of, any known greedy strategy?This may be the case, if it turns out that it is always preferable to introduce a new node rather than revise an old node's weights.Or, with appropriate regularization, would such a method instead converge to a tree with a limited number of nodes (as Fig. 2 might suggest)?I think this issue cuts to the core of what makes the proposed method worthwhile compared to naive DTs, and deserves perhaps more careful analysis than is currently offered by the paper.For instance, a critical issue left ambiguous is this: what happens if some node's weight is equal to zero?The discussion on page 3 implies that we take the right branch, which seems like a dubious choice.It seems like it would be preferable to output a decision at this node; later, we might decide to make this weight nonzero, corresponding to splitting the node.Precisely defining what happens here is important, because it has a direct analog to pruning strategies for greedy training.I have one significant practical concern regarding the motivation for the work.Namely, I would say that plain DTs are useful mainly to the extent that they are interpretable; if the interpretability requirement is dropped, then it is usually better to use other methods, which generally offer much better performance.As stated, I would expect the method to produce non-interpretable results, since the L2 regularization would produce dense weight vectors.Although switching to L1 regularization might help, it is possible that performance would drop if we were forced to regularize to the point where exactly 1-sparse splits were produced.So, this would definitely need to be tested.My other main concern is that the experiments don't go into enough detail regarding other baselines.No details are given as to which pruning strategies were employed for the naive approach.Since a range of regularization parameters were tried for the proposed method, I would expect the naive method to be similarly tried with a few different pruning strategies and/or parameters.I think it would also be fair to try some other trivial strategies for global optimization, such as training a greedy method with random subsets of the training data, and choosing the one that minimizes the desired loss on the full training or validation set.Regarding clarity, the paper is reasonably well-written. My only comment is that the parts describing inference could be a bit clearer.Section 6.2, for example, could be summarized by saying that the objective is first maximized for all g corresponding to a given leaf, which is then maximized over all leaves.Section 6.4 could also be a little more explicit.In summary, I think this is a very clever approach that raises some very interesting questions.However, it is unclear whether this is a practically useful method at this point, both due to the aforementioned interpretability issue (why use plain DTs in the first place, if they are not interpretable?) and due to a lack of details in the experiments.PS: although the title mentions forests, the paper does not address this case.POST-REBUTTAL COMMENTSI still think that the case of zero weights is a critical case that deserves further analysis, for the reasons I brought up in my review.Reading the rebuttal, it sounds like the weights are probably initialized randomly, which is why this issue doesn't seem to come up in practice.However, I have a feeling that what would really make this method interesting would be the case where the weights are initially zero and are gradually increased.I encourage the authors to consider this direction.