Understanding variable importances in forests of randomized trees
Despite growing interest and practical use in various scientific areas, variable importances derived from tree-based ensemble methods are not well understood from a theoretical point of view. In this work we characterize the Mean Decrease Impurity (MDI) variable importances as measured by an ensemble of totally randomized trees in asymptotic sample and ensemble size conditions. We derive a three-level decomposition of the information jointly provided by all input variables about the output in terms of i) the MDI importance of each input variable, ii) the degree of interaction of a given input variable with the other input variables, iii) the different interaction terms of a given degree. We then show that this MDI importance of a variable is equal to zero if and only if the variable is irrelevant and that the MDI importance of a relevant variable is invariant with respect to the removal or the addition of irrelevant variables. We illustrate these properties on a simple example and discuss how they may change in the case of non-totally randomized trees such as Random Forests and Extra-Trees.
Variable importance measures are often used to 
      highlight key predictor variables in tree-based ensemble models. However, 
      there has generally been a lack of a theoretical understanding of these 
      measures. In this paper, the authors study the theoretical properties of 
      the Mean Decrease Impurity (MDI) importance measures (such as the Gini 
      importance). Through most of the paper they use the Shanon entropy as the 
      impurity measure but show that the theorems and results are applicable to 
      any impurity measure. They begin with an asymptotic analysis of totally 
      randomized fully-developed tree ensembles learned using an infinitely 
      large ensemble. They show that under these conditions the variable 
      importance for all variables provide a three-level decomposition of the 
      information contained in the set of input variables about the output 
      variable. First, they show that the information decomposes as a sum of 
      importance scores of all input variables. Second, the importance of each 
      variable decomposes as a sum over all degrees of interactions of that 
      variable with other variables. The third level of decomposition for each 
      degree is over all possible combinatorial interactions of that degree. 
      Next, the authors show that that the MDI importance of a variable is equal 
      to zero if and only if the variable is irrelevant and that the MDI 
      importance of a relevant variable is invariant with respect to the removal 
      or the addition of irrelevant variables. The authors then analyze the 
      properties of the variable importance measures in depth-pruned ensembles 
      of randomized trees. They show that as long as the pruning depth is 
      greater than the total number of true relevant variables, these relevant 
      variables still obtain a strictly positive importance score (albeit 
      different from their values in fully-developed trees) and the irrelevant 
      variables continue to obtain an importance of zero. Finally, for the case 
      of ensembles consisting of non-totally randomized trees such as random 
      forests, where top scoring predictors are selected at splits from a random 
      subset of predictors, they show strong masking effects by the strongest 
      variables resulting in over or under-estimation of importance scores. They 
      finally show the relevance of their results through a simple yet 
      insightful example of digit recognition. The paper is very 
      accessible and written in a very understandable manner. At the same time, 
      the theorems and results discussed are very interesting and provide some 
      nice theoretical insights into variable importance measures from idealized 
      asymptotic models to more realistic models. The masking effects by the 
      strongest predictors and the inability of the variable importance to 
      appropriately account for interactions and dependencies between variables 
      for random-forest like models are very relevant to practical analyses and 
      may be amplified even more in cases involving correlated or partially 
      correlated variables. I think the paper lays down some nice initial 
      results and a general framework to analyze more realistic models in the 
      future and potentially point to better strategies for learning trees or 
      computing more accurate importance scores. The paper will definitely be of 
      interest to the general machine learning 
      community.