In this paper, a deep boosting algorithm is developed to learn more discriminative ensemble classifier by seamlessly combining a set of base deep CNNs (base experts) with diverse capabilities, e.g., these base deep CNNs are sequentially trained to recognize a set of object classes in an easy-to-hard way according to their learning complexities. Our experimental results have demonstrated that our deep boosting algorithm can significantly improve the accuracy rates on largescale visual recognition.
This paper consider a version of boosting where in each iteration only class weights are updated rather than sample weights and apply that to a series of CNNs for object recognition tasks.
While the paper is comprehensive in their derivations (very similar to original boosting papers and in many cases one to one translation of derivations), it lacks addressing a few fundamental questions:
- AdaBoost optimises exponential loss function via functional gradient descent in the space of weak learners. It's not clear what kind of loss function is really being optimised here. It feels like it should be the same, but the tweaks applied to fix weights across all samples for a class doesn't make it not clear what is that really gets optimised at the end.
- While the motivation is that classes have different complexities to learn and hence you might want each base model to focus on different classes, it is not clear why this methods should be better than normal boosting: if a class is more difficult, it's expected that their samples will have higher weights and hence the next base model will focus more on them. And crudely speaking, you can think of a class weight to be the expectation of its sample weights and you will end up in a similar setup.
- Choice of using large CNNs as base models for boosting isn't appealing in practical terms, such models will give you the ability to have only a few iterations and hence you can't achieve any convergence that often is the target of boosting models with many base learners.
- Experimentally, paper would benefit with better comparisons and studies: 1) state-of-the-art methods haven't been compared against (e.g. ImageNet experiment compares to 2 years old method) 2) comparisons to using normal AdaBoost on more complex methods haven't been studied (other than the MNIST) 3) comparison to simply ensembling with random initialisations.
Other comments:
- Paper would benefit from writing improvements to make it read better.
- "simply use the weighted error function": I don't think this is correct, AdaBoost loss function is an exponential loss. When you train the base learners, their loss functions will become weighted.
-  "to replace the softmax error function (used in deep learning)": I don't think we have softmax error function