  Dropout and other feature noising schemes have shown promising results in
controlling over-fitting by artificially corrupting the training data. Though
extensive theoretical and empirical studies have been performed for generalized
linear models, little work has been done for support vector machines (SVMs),
one of the most successful approaches for supervised learning. This paper
presents dropout training for linear SVMs. To deal with the intractable
expectation of the non-smooth hinge loss under corrupting distributions, we
develop an iteratively re-weighted least square (IRLS) algorithm by exploring
data augmentation techniques. Our algorithm iteratively minimizes the
expectation of a re-weighted least square problem, where the re-weights have
closed-form solutions. The similar ideas are applied to develop a new IRLS
algorithm for the expected logistic loss under corrupting distributions. Our
algorithms offer insights on the connection and difference between the hinge
loss and logistic loss in dropout training. Empirical results on several real
datasets demonstrate the effectiveness of dropout training on significantly
boosting the classification accuracy of linear SVMs.

Machine Learning