Large Margin Discriminant Dimensionality Reduction in Prediction Space
In this paper we establish a duality between boosting and SVM, and use this to derive a novel discriminant dimensionality reduction algorithm. In particular, using the multiclass formulation of boosting and SVM we note that both use a combination of mapping and linear classification to maximize the multiclass margin. In SVM this is implemented using a pre-defined mapping (induced by the kernel) and optimizing the linear classifiers. In boosting the linear classifiers are pre-defined and the mapping (predictor) is learned through a combination of weak learners. We argue that the intermediate mapping, i.e. boosting predictor, is preserving the discriminant aspects of the data and that by controlling the dimension of this mapping it is possible to obtain discriminant low dimensional representations for the data. We use the aforementioned duality and propose a new method, Large Margin Discriminant Dimensionality Reduction (LADDER) that jointly learns the mapping and the linear classifiers in an efficient manner. This leads to a data-driven mapping which can embed data into any number of dimensions. Experimental results show that this embedding can significantly improve performance on tasks such as hashing and image/scene classification.
This paper deals with two major themes. First it describes a duality between multi-class SVMs and multiclass adaboost. Secondly it uses this duality to derive a novel dimensionality reduction method that preserves discriminability between different classes.
The authors modify the MCBoost criterion, in order to allow for multi-class boosting that is based on arbitrary number of dimensions (compared to a previous formulation that limits the number of dimensions to the number of classes). This lift of the limits in terms of dimensionality allows for a boosting-like framework that is comprised of controllable amount of boosting functions, and thus can be used as.  + The connection between MC-Boost and MV-SVM is interesting, and the discussion is good. The discussion of the differences in terms of the predictor and the codewords (e.g. table 1) is also quite readable. Is the fact that both MC-SVM and MC-Boost try to maximise the margin well known?  + The authors present improved results in terms of error rate, and in terms of mAP. More specifically, LADDER outperforms MCBoost by 2% in terms of classification accuracy. In addition, LADDER outperforms MCBoost for across all number of dimensions in terms of error rate (Figure 3).  - More details could be given in terms of dimensionality reduction and CNN. This is a promising line of work, and has been shown to outperform all the other techniques even for cases where the size training data is not huge as the authors suggest. (e.g. DrLim Hadsell et al 2006). This holds for the literature review.  - Results could be compared with the new results based on CNNs. The representations learned on CNNs can be quite powerful, since both the features and the discriminant projection is learned. In addition, altough CNN's are highly non-convex, in practice they give better results than convex methods. The only comparison with CNNs is given in terms of using the features extracted from conv5 (Table 2) in order to compare with the BoW features. However, the real power of deep learning comes from simultaneous learning of both the features and the discriminative projection functions.  - Another convex method presented by Simonyan,Vedaldi and Zisserman (PAMI 2014) is not compared. In fact the authors only compare with lower accuracy methods such as PCA or LDA and their kernel equivalents.  - In general the experimental evaluation is not convincing, and there are many more experiments needed to show the merits of the proposed method. Convergence characteristics and more sub-analyses of the proposed method are needed.