Online Multiclass Boosting
Recent work has extended the theoretical analysis of boosting algorithms to multiclass problems and to online settings. However, the multiclass extension is in the batch setting and the online extensions only consider binary classification. We fill this gap in the literature by defining, and justifying, a weak learning condition for online multiclass boosting. This condition leads to an optimal boosting algorithm that requires the minimal number of weak learners to achieve a certain accuracy. Additionally, we propose an adaptive algorithm which is near optimal and enjoys an excellent performance on real data due to its adaptive property.
In this paper the authors introduce two learning algorithms dealing with online learning in a multi-class setting. The novelty resides in combining a multi-class boosting framework providing solid theoretical results with an online learning framework. Notable contributions of the paper include: a weak learning condition for multi-class online learning, a general boosting algorithm and a more practical version. I have two main concerns about this paper.
The paper is overall well written, the key ideas are ordered in an easy to follow manner and, as far as I can tell, the theoretical results are sound. However, at times the paper is hard to follow without switching over to the supplementary material. For instance, the link between the weak learning condition in batch learning and the one presented in the paper, is not obvious without reading the supplementary material; Section 3.2 is also hard to follow since the supplementary material is needed in order to grasp key ideas. While I do understand the authors' eagerness to include most of their work in the paper, in my opinion, the main paper should be as close as possible a stand alone paper, which isn't the case for this paper.
My second concern is related to the experimental and (the missing) discussion sections. The experimental results suggest that the proposed methods can achieve better results than other online methods, however their run time is obviously far more important. As such two questions arise : 1) Is there a case where the proposed methods are more interesting to use than existing ones wrt the ratio performance/run time? 2) Is it possible to speed up the proposed methods without significant loss in their performances? A discussion section treating both questions would've been appreciated. I also think that more experimental results are needed in order to evaluate how the proposed methods scale wrt the number of classes and examples.
When introducing the adaptive algorithm the authors propose to discard the weak learning condition and introduce an empirical edge. However the empirical edge takes its values in [-1,1], while the edge in the weak learning condition takes its values in (0,1). In practice, how is a negative edge dealt with? Is the training sample discarded?