Logarithmic Time Online Multiclass prediction
We study the problem of multiclass classification with an extremely large number of classes (k), with the goal of obtaining train and test time complexity logarithmic in the number of classes. We develop top-down tree construction approaches for constructing logarithmic depth trees. On the theoretical front, we formulate a new objective function, which is optimized at each node of the tree and creates dynamic partitions of the data which are both pure (in terms of class labels) and balanced. We demonstrate that under favorable conditions, we can construct logarithmic depth trees that have leaves with low label entropy. However, the objective function at the nodes is challenging to optimize computationally. We address the empirical problem with a new online decision tree construction procedure. Experiments demonstrate that this online algorithm quickly achieves improvement in test error compared to more common logarithmic training time approaches, which makes it a plausible method in computationally constrained large-k applications.
Heavy reviewSummary: The paper concerns multi-class problems with a large number of classes. It introduces a novel label tree classifier that learns and predicts in logarithmic time in the number of classes. Theoretical guarantees in terms of a boosting-like theorem have been proven. Moreover, not only node classifiers, but also the structure of the tree is trained online. Additionally, the authors show a simple subtree swapping procedure that ensures proper balancing of the tree. Empirical results show attractiveness of the introduced approach.Quality:The paper presents both theoretical and empirical results. The method is clearly derived and theoretically justified. Some minor comments are given below.The objective after reformulation can be stated as:J(h) = 2 \sum_{i=1}^k | \Pr(i) \Pr(h(x) > 0) - \Pr(i, h(x) > 0) |This is a kind of an independence test between i and h(x) > 0. I suppose that such measures have been used in rule induction (e.g., in generation of association rules) and decision tree learning. This is also similar to the so-called Carnap's confirmation measure (see, "Comparison of confirmation measures", Cognition, 2007, and "Logical foundations of probability (2nd ed.).", 1962).Decision tree learning can be introduced as a recurrent procedure of empirical risk minimization guided by a surrogate loss function, where both splits and leaf predictions are computed as empirical risk minimizers (the notion of a purity measure is not needed then). For C4.5, this is logistic loss, and for CART squared error loss. It would be interesting to check whether there exists a loss function that corresponds to J(h).The authors do not clearly describe the difference between LOMTree and standard decision trees. Footnote 4 states that standard criteria such as Shannon or Gini entropy satisfy requirement of balanced and pure splits. Since it is possible to adapt standard decision trees to the problems with a large number of classes [16,17], it is not clear what is the difference between these two approaches. Can we substitute J(h) by Shannon entropy?I suppose that there exists a kind of a trade-off between test-time complexity and the complexity of the function class, i.e., by minimizing the number of prediction steps we should probably extend the complexity of the function class. This would explain the empirical results of OAA and LOMTree. Could you comment on this?The prediction time results are given per test example. The overall method is also designed for improving test time for a single test example. However, when test examples are coming in mini-batches different tricks can be used to improve test time. How to use this method in such scenario? Clarity:The paper is clearly written. Originality:The paper is very original as it clearly states the problem of efficient learning and prediction in case of large output spaces. The introduced algorithm is also novel, however, the authors should take into account that the problem of decision tree induction was exhaustively studied in 80' and 90'. Many algorithms have already been introduced that share similarities with LOMtrees (incremental learning of trees, trees for data streams, Hoeffding trees, trees with linear splits). Significance:The paper will be (already is) very influential for large-scale machine learning.After rebuttal:Thank you for your responses. I gave the references to confirmation measures since this was the most similar measure I knew to your objective. It would be great if you will explore this topic in the extended (e.g., arxiv or journal) version of the article. I have found your objective very interesting, but I think that similar measures have already been considered in many settings and many interesting links can be found. I agree that such measures are not so popular and successful in decision tree learning, especially in the multi-class setting.The same concerns incremental learning of decision trees. There are some approaches that you can easily find in Internet. The question is how they relate to your approach. One well-know approach are Very Fast Decision Trees (aka Hoeffding trees) introduced by Domingos and Hulten (2000). A detailed discussion can be postponed however to the extended version of your paper.