Multi-Layered Gradient Boosting Decision Trees
Multi-layered distributed representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are still the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical distributed representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive backpropagation nor differentiability. Experiments confirmed the effectiveness of the model in terms of performance and representation learning ability.
Short overview: Authors propose to build a neural network using gradient boosted trees as components in the layers. To train such a structure, since the gbdts are not able to propagate the gradient, they propose to use a method inspired by the target propagation: each gradient boosted trees is built to approximate a gradient of loss of prediction function and a pseudo target, with respect to the prediction function. Pseudo targets are updated at each iteration using the reverse mapping of the built tree representation and the pseudo label of the next layer. The reverse mapping can be found using the reconstruction loss. At each iteration, each layer's ensemble grows by one boosting tree Authors hint at potential applications of blocking adversarial attacks, that rely on estimating the gradients of the final loss with respect to input, which would not work for layers that can't propagate the gradients, however this direction is not explored in this paper  Detailed comments: Overall, an interesting idea of co-training gbdts with nns. The idea is not new, but discarding backprop and using target propagation, that allows gbdts to do what they are best for - e.g. build trees to simulate gradients - is neat and witty.   Overall, well written and well structured, but I do have some points of confusion after reading it:  a) Pseudo labels are all computed from top to bottom - i am not sure where you think is the top and where the bottom is. is it to say that the first pseudo label is from the last (output layer) and then you go backward to the input layer. If yes, this top to bottom needs to be rephrased. Some people draw neural nets horizontally actually... (you included, figure 1!), but even if you draw vertically, the output is the bottom, and you go from the bottom to the top  b) What is the structure of your pseudo-inverse function G_i^t? Is it essentially a matrix of dimension R^d_i x R^d_(i-1)?  In your algorithm, I am not sure why you fit the regression tree and update the inverse mapping interchangeably for a number of iterations? Why not to first minimize the loss (using the tree from the previous iterations) to obtain the inverse mapping (so for a number of iterations, do a gradient descent update using the reconstruction loss) and then when it is settled, build a tree using the inverse loss and found inverse mapping, which can be used to get pseudo label at this point. (i am referring to the for loop k=1..K). Instead to update the G you make a pseudo gradient descent step using the built tree, which does not seem very intuitive to me. Plus it also seems that you discard all the trees built at this step (you use them just to update G)  c )As far as i remember, target propagation works slower than backprop, I am not sure if you observed the same   My main concern is the experiments and their reproducibility: the way they are described, it is not clear what the actual setup was used for their method, and if the comparison is fair (were all the competing models tuned?). I am having hard time believing that well tuned 2-hidden layer nn would underperform so badly. What do you mean for a setup of 5-3 for instance? AT first i thought those were the number of neurons in nn, but on the second thought, i think you mean 3 is the dimension of the first tree layer, 5 of the second. But how many trees are in each layer? This really needs to be clarified.  Why accuracy and reconstruction loss of is not reported for the synthetic datasets. As well as metrics of at least one competing baselines, like just a nn? How are those 128-128 setups are chosen for income prediction? i think the fact that nns are trained with exactly the same structure (128-128 for a hidden layer) means you have a huge overparameterization, that's why they don't do well. The structure of the net should be just tuned to provide a fair point of comparison. The same as hyperparameters for xgboost (eg learning rate of 0.3 is default and is large, income is a simple dataset, and i expect that you just overfit at this point)   Additionally, I not sure what the exact target application is? The adversarial learning is interesting, but it was not explored, and apart form this, i am struggling to imagine where an ensemble per layer should be more efficient than a tuned hidden layer.  Suggestions: to facilitate the reading, it would be nice to add a short section on target propagation, since it is a backbone of your method, to facilitate the reading. You seem to have enough space for that  Nits: line 185: the experiments ... is->are line 187: where did CNNs came from? i think even ur synthetic experiments dont use the convolutional layer (but i am not sure). Income prediction surely does not Table 1: are the diffs significant on 16k test dataset for income? line 270: interesting to integrate line 274: does not make  I read authors  rebuttal and I they addressed the majority of questions I had