Large-Margin Convex Polytope Machine
We present the Convex Polytope Machine (CPM), a novel non-linear learning algorithm for large-scale binary classification tasks. The CPM finds a large margin convex polytope separator which encloses one class. We develop a stochastic gradient descent based algorithm that is amenable to massive datasets, and augment it with a heuristic procedure to avoid sub-optimal local minima. Our experimental evaluations of the CPM on large-scale datasets from distinct domains (MNIST handwritten digit recognition, text topic, and web security) demonstrate that the CPM trains models faster, sometimes several orders of magnitude, than state-ofthe-art similar approaches and kernel-SVM methods while achieving comparable or better classification performance. Our empirical results suggest that, unlike prior similar approaches, we do not need to control the number of sub-classifiers (sides of the polytope) to avoid overfitting.
This paper presents a large-margin learning algorithm for convex polytope machines for binary classification tasks. The optimization objective is obtained from relaxing the problem of maximizing the total margin. A preliminary generalization bound regarding the objective function is presented. The paper proposes to use SGD to solve the optimization problem Eq. (6).  The experiments, which are conducted on standard benchmarks, demonstrate convincing performance over baselines including RBF SVM. It seems that the proposed formulation of optimization objective could be further clarified or improved. The paper claims to optimize the total margin (line 170-176). Indeed, the first objectives (Eq. (1) and Eq. (2)) directly maximize the total margin. These objective functions are non-convex, as pointed out in the paper. Therefore, the authors propose Eq. (3-5) and Eq. (6) as the objective function to be optimized, suggesting that they are the relaxation of Eq. (1) and Eq. (2).  However, the quantitative connection between Eq. (3-6) and Eq. (1-2) is unclear to me and is unfortunately not clarified in the paper.  It would be better if the paper develops or explains in more quantitative way how minimizing Eq. (3-5) will maximize the total margin. Furthermore, even after relaxation, the objective Eq. (6) is still non-convex. Thus I doubt whether the proposed relaxation has served its purpose well. In sum, the proposed objective function Eq. (6) seem to be a choice that is not well-justified, since (1) it is unclear how well it maximizes the total margin; (2) the relaxed objective is still non-convex.The main theoretical result (Theorem 1) in the paper is proven using standard arguments of proving generalization bounds. The proof does not seem to contain much new techniques or insights; it would be better if the paper points out the novelties of its proof, if any. The proposed SGD algorithm is heuristic and lacks theoretical underpinnings. The paper suggests that pure SGD does not work (due to the non-convexity) and therefore the authors add an entropy term to encourage smaller clusters. However, besides some intuitions, there are no formal analyses on how this entropy term will influence the objective function. Further, the paper does not describe clearly how the entropy is computed; line 278 says that it is computed by assuming x~D^+, but I cannot find the definition of _distribution_ D^+ in the text. While these details are missing or unclear, the paper includes some implementation details (which are very standard and well-known techniques) that add little value to the paper, e.g., maintaining scaling factors, using dense/sparse matrices, the internal data structure and using C++11. I think that the space may be better utilized if it is used to present more algorithm/model related details. Nevertheless, I appreciate the authors’ promise of releasing the source code to the community.The experiment results shown in the paper are good. But many experimental results are omitted (including the influence of K on the rest of data sets) due to space constraints. These results (especially the influence of K) may be useful of understanding the advantages of large-margin convex polytope machines over ordinary SVMs. Perhaps the space of supplementary material can be utilized to present more experiments results.The writing of this paper could be improved in a few places. - line 248. “sparse high dimensional space”; - line 265-266. “a pure SGD approach could get stuck in a local optimum. We found that pure SGD gets stuck in low-quality local optima in practice”;