Multi-class SVMs: From Tighter Data-Dependent Generalization Bounds to Novel Algorithms
This paper studies the generalization performance of multi-class classification algorithms, for which we obtain-for the first time-a data-dependent generalization error bound with a logarithmic dependence on the class size, substantially improving the state-of-the-art linear dependence in the existing data-dependent generalization analysis. The theoretical analysis motivates us to introduce a new multi-class classification machine based on p -norm regularization, where the parameter p controls the complexity of the corresponding bounds. We derive an efficient optimization algorithm based on Fenchel duality theory. Benchmarks on several real-world datasets show that the proposed algorithm can achieve significant accuracy gains over the state of the art.
The paper describes a family of multi-class SVMs parameterised by a single parameter p. The family contains the Crammer & Singer SVM as a special case.Corresponding generalization bounds are derived, which for certain parameter choices have a logarithmic dependency on the number of classes. The latter has - to my knowledge - not been shown for any multi-class classifier.The corresponding learning machine is derived including its Fenchel dual problem. There is an empirical evaluation, which shows that the approach works. However, the experiments are the weak spot of the paper.Only three test problems are considered. It is not clear which kernel function is used, therefore I assume a linear kernel. The Crammer & Singer SVM serves as the baseline, the parameter p is optimised on a fine grid (including the value p=2, which gives the Crammer & Singer SVM). The results for the best p value are reported (optimised for each data set independently), which are slightly better than the Crammer & Singer SVM results.That the Crammer & Singer SVM results are achieved does not come as a surprise, as p=2 is tested for the new machines, i.e., the new machines include the Crammer & Singer SVM. The grid for optimising p is very fine. That is, the best of many values of p (i.e., many different machines) are compared to the Crammer & Singer variant. That one always find one machine for each data set is not surprising, actually I think that the improvements are so small is rather disappointing. What if the model selection for the Crammer & Singer variant is refined such that in the end as many machines are tested as in the case when also p is optimised? For example, using a finer grid for regularisation parameter and kernel bandwidth parameter (in case of a Gaussian kernel SVM). Better statistics about the robustness w.r.t. p should be reported.Testing more data sets with a linear as well as a non-linear kernel is mandatory.Computation times for solving the optimisation problems are not given. The general problem of the new machines is more difficult to solve than the Crammer & Singer QP, and the scaling of the optimisation time w.r.t. the number of training data points is important to see. Timing experiments are a must.Long training times (which I assume) and the necessity to adapt an additional hyperparameter reduce the practical value of the proposed approach.Anyway, the theoretical result is strong.  Minor comments: mercer -> Mercer lines 394,395: The description of the grid for p does not seem to be consistent. 1.2 and 1.25 differ by 0.5 - how is this related to the step sizes 0.5 and 0.1?