Robust Logistic Regression and Classification
We consider logistic regression with arbitrary outliers in the covariate matrix. We propose a new robust logistic regression algorithm, called RoLR, that estimates the parameter through a simple linear programming procedure. We prove that RoLR is robust to a constant fraction of adversarial outliers. To the best of our knowledge, this is the first result on estimating logistic regression model when the covariate matrix is corrupted with any performance guarantees. Besides regression, we apply RoLR to solving binary classification problems where a fraction of training samples are corrupted.
The manuscript proposes a new method for robust logistic regression with a focus on dealing with outliers with high leverage. Outliers are assumed to come from an arbitrary and unknown distribution, but the number of outliers is assumed to be known ahead of time, which is a theoretically convenient, but practically slightly more troubling assumption.The method is based on maximising the sum of y*x'beta, but only summed over the observations which contribute in absolute terms the least to the objective function, thus outliers with large leverage are excluded. This can be elegantly translated into a linear programming problem.The authors derive some risk bounds. In common with most of these bounds, they are probably too loose to have a practical quantitative use, but are useful for qualitative interpretation.The manuscript contains a small simulation study in which the proposed method compares very favourably with classical logistic regression.A few more detailed comments:- Why do you need the preprocessing step? Observations with large ||x|| should also be among the observations with large y*x'beta and thus omitted by the algorithm. Also T is decreasing in n (and tends to 0 in the limit), so the more data we have, the more observations are thrown out by the preprocessing. For example, what proportion of observations is removed by the preprocessing step in the simulation?Also, I am surprised that T is not chosen based on n or lambda. - On a related point, it might be worth looking at how well logistic regression does after the same preprocessing is carried out.- Given the preprocessing and the set-up of the method, situations where (like in the simulation shown) the outliers have much larger variance in the covariates are quite favourable to the proposed method (they are very likely to trigger the preprocessing criterion or lead to a large contribution to the objective function and are thus omitted). What happens is sigma_o is decreased, so that outliers look less like obvious outliers and are more likely to sneak into the n smallest contributions (and thus not excluded by the method)? Will the method perform better or worse?- One important question is how much worse the method performs when compared to logistic regression when there are no outliers. The very left end of figure 2 suggests no big difference in terms of estimating beta, but a more substantial difference when looking at the misclassification rate. Essentially, what price do we have to pay for robustness?- In practice, how do I choose n? How sensitive is the method to the choice of n?The manuscript is well written and clearly structured.