Multi-Class Deep Boosting
We present new ensemble learning algorithms for multi-class classification. Our algorithms can use as a base classifier set a family of deep decision trees or other rich or complex families and yet benefit from strong generalization guarantees. We give new data-dependent learning bounds for convex ensembles in the multiclass classification setting expressed in terms of the Rademacher complexities of the sub-families composing the base classifier set, and the mixture weight assigned to each sub-family. These bounds are finer than existing ones both thanks to an improved dependency on the number of classes and, more crucially, by virtue of a more favorable complexity term expressed as an average of the Rademacher complexities based on the ensemble's mixture weights. We introduce and discuss several new multi-class ensemble algorithms benefiting from these guarantees, prove positive results for the H-consistency of several of them, and report the results of experiments showing that their performance compares favorably with that of multi-class versions of AdaBoost and Logistic Regression and their L 1 -regularized counterparts.
Summary of the paper: The paper introduces a new class of multi-class boosting algorithms with base learners that are regularized based on their Rademacher complexities. The experiments show great improvement over other multi-class approaches in the literature and on the non-regularized version of the proposed algorithm.Detailed remarks:For the title: in general, the term "deep" in deep learning is used for saying that the representation is hierarchical in a compositional sense. Trees are obviously not deep in this sense, and so just because you are using large trees in boosting (a quite common setup in practice), it will not make the approach "deep". You still use a single "hidden layer" of trees, combined linearly, this is definitely a shallow architecture. Thus I strongly object to the title, it is misleading.Lines 042-043: In practice, boosted trees almost always outperform boosted stumps \cite{CaNi06, Keg14}, and when the validated tree sizes are obtained by proper hyper-parameter optimization, they can be quite large, the same order as found in your experiments. Moreover, when boosting multi-class Hamming trees in AB.MH, \cite{Keg14} also found that on most of the data sets there is very little overfitting, basically one can boost trees of several tens of inner nodes for ten thousand iterations (see, for example, pendigits or letters, two sets on which most algorithms are tested), without increasing the test error. So, the statement of "boosting has been observed to overfit in practice", derived from 15 year-old papers, should be revised. Sometimes it overfits, sometimes it doesn't, and basically we don't know when it does and why it does when it does.Lines 046-047: To my knowledge, the first paper proposing adaptive regularization of base classifiers is \cite{KeWa04}. The intuitive idea is the same and the final algorithm is not that different either (coefficients have to be shrunk by a quantity related to the empirical complexity of the weak classifier).Although we cannot expect that a conference paper surveys all multiclass boosting algorithms, the paper should at least mention those that seem to be state of the art: AOSO \cite{SuReZh12}, ABC \cite{Li09,Li09a}, Gao-Koller's iterative weak learner in hinge-boost \cite{GaKo11}, and AB.MH with Hamming trees \cite{Keg14} (it seems to me that the trees in this latter are quite similar to those used in this submission).The experimental setup. I was honestly stunned reading this: "We recorded the parameter tuple that had the lowest average error across all 10 runs, and this average error and the standard deviation of the error is reported in Table 1 and Table 2, along with the average number of trees and the average size of the trees in the ensembles." You're validating on the test set, something that we teach to our students never to do. The consequence is twofold. First, I cannot compare the errors to those available in the literature. Some of the errors (e.g. on pendigit) looked suspiciously low, 3-4 times lower than I've ever seen, that's when I started to check your experimental setup. Second, the empirical comparison of the algorithms you tested is tainted. It is obvious that if you take an algorithm and you add hyper-parameters (the AB.MR -> L1 AB.MR -> MDeepBoostSum chain), the minimum test error can only decrease. The ballpark range of the "improvements" is very much in line with this view: you simply harvested the fact that the minimum of a larger sample is smaller than the minimum of a smaller sample, even if they come from the same distribution.Now, I know that this seems like a detail for a theoretician, but for people using these algorithms what you are claiming is important. We have tested a lot of ways of regularizing the weak learners up to about ten years ago (you referred to some of the works), it never worked, more precisely, we didn't seem to need it. There were some indications that it could help on small data sets \cite{KeWa04}, but, exactly because of the small size of the sets, results were inconclusive. If you now claim that it is not the case, the experimental validation has to be rock solid. My suggestion is that you make an attempt to redo the experiments doing proper double cross validation during the rebuttal period, and show us the new results. If they are non-conclusive (that is, the regularized version doesn't beat the standard algorithm), I would say the paper could still be accepted, but the message has to be altered to something like "here is an interesting-looking algorithm with some strong theoretical justifications, but regularization doesn't work".Providing an open source implementation of the algorithm would be a great way to make the experiments reproducible and to let people use the proposed techniques and to build on them.Pseudocode: for those who would like to implement the method starting from the pseudocode, it would be helpful to point towards the definitions of the quantities in the caption: \Lambda and S_t for t = 1 are undefined.@inproceedings{KeWa04,	Address = {Vancouver, Canada},	Author = {K\'{e}gl, B. and Wang, L.},	Booktitle = {Advances in Neural Information Processing Systems},	Pages = {665--672},	Publisher = {The MIT Press},	Title = {Boosting on manifolds: adaptive regularization of base classifiers},	Volume = {17},	Year = {2004}}@inproceedings{CaNi06,	Author = {Caruana, R. and Niculescu-Mizil, A.},	Booktitle = {Proceedings of the 23rd International Conference on Machine Learning},	Pages = {161--168},	Title = {An Empirical Comparison of Supervised Learning Algorithms},	Year = {2006}}@inproceedings{KeBu09,	Address = {Montreal, Canada},	Author = {K\'{e}gl, B. and Busa-Fekete, R.},	Booktitle = {International Conference on Machine Learning},	Pages = {497--504},	Title = {Boosting products of base classifiers},	Volume = {26},	Year = {2009}}@inproceedings{Li09,	Author = {Li, P.},	Booktitle = {International Conference on Machine Learning},	Title = {{ABC}-{B}oost: Adaptive Base Class Boost for Multi-class Classification},	Year = {2009}}@techreport{Li09a,	Author = {Li, P.},	Institution = {Arxiv preprint},	Number = {arXiv:0908.4144},	Title = {{ABC-LogitBoost} for Multi-class Classification},	Year = {2009}}@inproceedings{GaKo11,	Author = {Gao, T. and Koller, D.},	Booktitle = {International Conference on Machine Learning},	Title = {Multiclass boosting with hinge loss based on output coding},	Year = {2011}}@inproceedings{SuReZh12,	Author = {Sun, P. and Reid, M. D. and Zhou, J.},	Booktitle = {International Conference on Machine Learning (ICML)},	Title = {{AOSO-LogitBoost}: Adaptive One-Vs-One {LogitBoost} for Multi-Class Problem},	Year = {2012}}@inproceedings{Keg14,	Abstract = { We train vector-valued decision trees within the framework of AdaBoost.MH. The key element of the method is a vector-valued decision stump, factorized into an input-independent vector of length $K$ and label-independent scalar classifier.},	Author = {K\'{e}gl, B.},	Booktitle = {International Conference on Learning Representations},	Title = {The return of {AdaBoost.MH}: multi-class {H}amming trees},	Url = {http://arxiv.org/abs/1312.6086},	Year = {2014}}----I looked at the new experimental results and indeed they are more reasonable now. As expected, errors increased significantly (eg., on letters and pendigits the errors increased by 6-7 standard deviations). In this light your answer "1) While it is of course possible to overestimate the performance of a learning algorithm by optimizing hyperparameters on the test set, this concern is less valid when the size of the test set is large relative to the "complexity" of hyperparameter space (as any generalization bound will attest). Note that our experiments varied only three hyperparameters over a large data set." looks strange. You just proved that my concern was highly relevant. If I look at your results now, I see no significant improvement by the added regularization which contradicts the main message of the paper (differences are in the 1-2 std range, completely compatible by statistical fluctuation). On the other hand, your results using AB.MR and AB.MR-L1 are really good, you do something in your tree-building procedure which seems to work better than what other people are doing (unfortunately, you don't give details on your procedure). I'm sympathizing with you in the sense that I know that simply describing a state-of-the-art algorithm without any theoretical results will never be accepted at NIPS, but still, at this point your theoretical results are irrelevant for a practitioner (it's not your theory that makes the algorithm good), and your practical results are irrelevant for a theoretician (for the same reason: it's not your practical results that make the theory interesting or relevant). It's two papers in one with a false conclusion. I'd be happy to accept the paper if you were honest about it.