Fast and Flexible Monotonic Functions with Ensembles of Lattices
For many machine learning problems, there are some inputs that are known to be positively (or negatively) related to the output, and in such cases training the model to respect that monotonic relationship can provide regularization, and makes the model more interpretable. However, flexible monotonic functions are computationally challenging to learn beyond a few features. We break through this barrier by learning ensembles of monotonic calibrated interpolated look-up tables (lattices). A key contribution is an automated algorithm for selecting feature subsets for the ensemble base models. We demonstrate that compared to random forests, these ensembles produce similar or better accuracy, while providing guaranteed monotonicity consistent with prior knowledge, smaller model size and faster evaluation.
The paper deals with learning monotonic functions. This can be useful for the data, for which there are some inputs that are known to be positively (or negatively) related to the output, and in such cases training the model to respect that monotonic relationship can provide regularization, and makes the model more interpretable. The main problem which is dealt in the paper is that flexible monotonic functions are computationally challenging to learn beyond a few features. The paper proposes to learn ensembles of monotonic calibrated interpolated look-up tables (lattices). A key contribution is an automated algorithm for selecting feature subsets for the ensemble base models. The algorithm is experimentally compared against random forests, and is shown to produce similar or better accuracy, while providing guaranteed monotonicity consistent with prior knowledge, smaller model size and faster evaluation.
The paper is heavily based on the previous work on lattice (and monotone lattice) regression, and is thus not self-contained. It is not even explained in the main part of the paper what is the actual function which a lattice represents (two examples are found in the supplementary materials, but I still found the explanation two brief). I think this should definitely be a part of the main paper, otherwise the reader is forced to check the previous work on this topic to understand what kind model is actually being considered.   The paper barely touches upon the computational complexity issues of optimizing (3), (7), and unnamed eq. in Step 3. While (7) is convex given calibrators being fixed, it is not clear whether it is convex when jointly optimized over calibration and lattice parameters (I doubt it is). Moreover, (3) seems highly complex and combinatorial optimization criterion. The optimization complexity of eq. is Step 3 is unclear to me (note that Step 3 requires knowing ensemble weights \alpha, which are only computed in Step 5; how is this issue solved?). The authors should clearly state which problems are easy to optimize, and which are only solved heuristically, likely returning suboptimal solution.  Given a complex training of lattice ensemble I missed the computational times of training. Since the authors report faster evaluation of their method over random forests at the testing phase, they should also report training time comparing to the competitors.  The experimental results are not very convincing: only around 1% of accuracy improvement is reported comparing to the random forests; and it is not clear if random forest is a good fit for these data sets. It also does not help that all data sets except the first one are proprietary (hard to find other experimental studies on these data sets). It would be beneficial to check at least one more state-of-the-art flexible regression method, such as, e.g., gradient boosting, on these data sets (the unconstrained lattice, as another competitor, is not enough, as it is not particularly known algorithm). For instance, xgboost (an implementation of gradient boosting) easily achieves over 86% accuracy on Adult, see https://jessesw.com/XG-Boost/. RuleFit algorithm of Friedman & Popescu (http://statweb.stanford.edu/~jhf/ftp/RuleFit.pdf) gives 13.6% test error (86.4% accuracy), etc. Apparently, they all beat both random forests and monotone lattices Adding more methods to the comparison would help to resolve the question, whether exploiting monotonicity actually gives an advantage in predictive performance.  Please explain what is novel in the paper comparing to Ref. [2]. Both papers share monotonic lattice regression, feature calibration and some of the optimization criteria (e.g., for joint optimization of calibration and lattice parameters). It looks as if the novel part is the ensemble learning of lattices, together with crystals algorithm to select features for lattice construction. And the parts related to incorporating monotonicity are not.