Bayesian Nonlinear Support Vector Machines and Discriminative Factor Modeling
A new Bayesian formulation is developed for nonlinear support vector machines (SVMs), based on a Gaussian process and with the SVM hinge loss expressed as a scaled mixture of normals. We then integrate the Bayesian SVM into a factor model, in which feature learning and nonlinear classifier design are performed jointly; almost all previous work on such discriminative feature learning has assumed a linear classifier. Inference is performed with expectation conditional maximization (ECM) and Markov Chain Monte Carlo (MCMC). An extensive set of experiments demonstrate the utility of using a nonlinear Bayesian SVM within discriminative feature learning and factor modeling, from the standpoints of accuracy and interpretability.
This paper mainly extends the Bayesian linear SVM in [5] to a nonlinear version, and further combines the Bayesian nonlinear SVM with factor models.The extension from linear to nonlinear however is quite trivial by simple adopting the standard kernel tricks. The resulted nonlinear version involves more complicated inference problem since it will also learn the kernel function parameters.The combination with factor models is produced by taking the two objectives together, while kernels are produced on the factor representations.There is not much novelity in terms of model extension and combination strategies. The overall learning problem is in fact a quite complicated non-convex optimization problem.Under the probabilistic Bayesian framework, some inference procedures are introduced to perform learning but there is no analysis about the complexity of the overall inference procedure.The experiments are limited to using Gaussian kernels. Is it possible to use other types of nonlinear kernels? Will it affect the inference algorithm?The datasets used in the experiments are too small (see Table 1). Large scale experiments need to be conducted.Moreover, the authors only compared the proposed approach to SVM and GPC methods.Considering the tasks addressed in this paper are simple binary classification tasks, why not compare to more advanced state-of-art methods?The authors motivate the work from the perspective of discriminative feature-learning models, which is a very general topic.I do not feel related works on this topic have been sufficiently discussed in the related work section.