Mondrian Forests: Efficient Online Random Forests
Ensembles of randomized decision trees, usually referred to as random forests, are widely used for classification and regression tasks in machine learning and statistics. Random forests achieve competitive predictive performance and are computationally efficient to train and test, making them excellent candidates for real-world prediction tasks. The most popular random forest variants (such as Breiman's random forest and extremely randomized trees) operate on batches of training data. Online methods are now in greater demand. Existing online random forests, however, require more training data than their batch counterpart to achieve comparable predictive performance. In this work, we use Mondrian processes (Roy and Teh, 2009) to construct ensembles of random decision trees we call Mondrian forests. Mondrian forests can be grown in an incremental/online fashion and remarkably, the distribution of online Mondrian forests is the same as that of batch Mondrian forests. Mondrian forests achieve competitive predictive performance comparable with existing online random forests and periodically retrained batch random forests, while being more than an order of magnitude faster, thus representing a better computation vs accuracy tradeoff.
In this paper, the authors introduce Mondrian forests, a classifier inspired by random forests, Mondrian processes, and interpolated Kneser-Ney (IKN). They define Mondrian trees (MT), show how a forest is built from such trees, discuss relations to Mondrian processes and hierarchical Pitman-Yor processes (HPYP). They describe how online training and prediction work. Finally, they show that Mondrian forests provide competitive predictive accuracy as a function of the amount of training data seen and some of the best predictive accuracy as a function of amount of time spent. They discuss briefly how this accuracy breaks down when too many useless features are input and how future work might ameliorate this issue.Overall the model in this paper is interesting, the experiments are convincing, and the application to streaming data in particular seems timely and important. I definitely would like to see this research published.My major qualm with this manuscript is that it reads as though the authors have tried to fit a journal paper into the space allotted by NIPS. As it is, the paper is too dense, and still too many statements are left undefined or unjustified. (It would make a great journal paper if all of these connections were filled in; I go over specific cases below.) That being said, I think this work would make an excellent conference submission if it were carefully pruned by focusing on only the core development needed to reach the experiments section. E.g. by: removing some of the excess tree description, simply defining the generative process for Mondrian trees (the authors might note the Mondrian process as inspiration without dwelling on the theoretical relations to Mondrian processes---at least not in the main text), defining and using IKN without all of the HPYP and NSP build-up (in particular since HPYP and NSP don't seem to be used in the experiments or for additional insight beyond the IKN connection), etc.Major points:* Line 064: Big O in probability is required here, rather than the existing deterministic Big O (cf. https://en.wikipedia.org/wiki/Big_O_in_probability_notation ). If the authors plan to keep this part as is, I recommend they say something brief about what O_p means as it is less standard in this community than Big O.* Line 078/079: That the distribution of the Mondrian tree is invariant to the order of data points is a statement that needs to be proved (with proof referenced here), at least in the supplementary material.* Line 081: The authors state the depth of the tree is "typically logarithmic in n." Elsewhere, they state complexity bounds for balanced trees. This depth statement needs to be justified if it remains in the manuscript. Ideally, the authors would demonstrate (with informative summaries and/or graphs and/or tables) how balanced the trees were that were found in the experimental runs. Again, this could happen in the supplementary material but should be referenced when it is used in the main text. If the trees are not uniformly balanced, it is worth providing an analysis for other cases as well, if possible.* Section 3: A long time is spent reviewing trees in general and rooted, binary trees in particular. This wouldn't be so bad except that short shrift is given to Mondrian processes and the definition of Mondrian trees. Readers in this audience are likely familiar with rooted, binary trees but are less likely to be familiar with Mondrian processes, which are not defined in this manuscript.* Line 159--163: Somewhere in this area it should be noted that one expects the model to be very sensitive to scaling (due to the role of the "linear dimension" here) and that the authors recommend scaling each dimension to [0,1]. Currently, [0,1]-scaling is just an offhand remark in the experiments section.* Line 169--174: Bayesian CART (Chipman, George, McCulloch 1998) should be mentioned here as well since variants of these ideas have already been used in that model.* Line 174--185: If mentioned, projectivity needs a proof, perhaps in the supplementary material. It's not immediately clear that the data dependence in MTs doesn't spoil projectivity but could be established within a few lines.* Line 203--207: If the authors keep the discussion of NSPs and PYPs, they should provide citations for these models when they are introduced. * Footnote 2: The stated relative tractability of NSPs needs a citation here.* Figure 2 caption: Half a page is too much text for a caption, which as a rule of thumb should not generally exceed one paragraph. The vast majority of this text should be in the main text---or at worst the supplementary material, despite the caption's appealing small font size.* Algorithm 4: It is worth noting somewhere in this area that the reason this extension works, at a high level, is the property of exponential distributions that: X ~ Exp(a), Y ~ Exp(b) implies min(X,Y) ~ Exp(a+b)* Section 6: Bayesian CART, and possibly related work such as BART, should appear at least briefly in the related work section.* Line 360: It would be good to see a citation for the stated popularity of this choice of M.* Line 361: Since \lambda = \infty in the experiments presumably implies that each data point is in its own leaf, this might be a chance to prune some discussion of \lambda from the manuscript and thereby shorten it. Alternatively, if the authors keep the discussion of \lambda, the authors should note the implications in the text of choosing \lambda = \infty.* Line 361: \gamma = 10 D: It would be helpful to use names for all parameters on this line instead of just symbols. Even better might be to provide a reference to where they were defined in the manuscript. \gamma is somewhat buried on line 210 as is.* Line 364: Why not run more random initializations and report error bars for each algorithm as well? If the error bars are so small as to not appear on the plot, the authors could mention this in the text or caption.Minor editing points:* Line 053: It seems like D_i should be (X_i, Y_i). The existing notation for D_{1:n} is clunky and doesn't degrade well for subsets of 1:n.* Line 106: "limit" -> "bounds"* Line 194: "encourages label" -> "encourages the label"* Line 194: "discuss the" -> "discuss how the"* Line 238: "rectngble" -> "rectangle"* Line 360: "As common" -> "As is common"