Distributionally Robust Logistic Regression
This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown datagenerating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.
This paper presents distributionally robust logistic regression (DRLR); the uncertainty set under consideration here is the Wasserstein ball of distributions, centered at the empirical (feature, label) distribution.The authors reformulate the problem setup so that it leads to a tractable convex optimization problem.The authors go on to provide a generalization error guarantee (which depends, counter-intuitively, on the assumption that the data-generating distribution is not heavy-tailed), and also present (tractable) convex optimization problems for computing upper and lower bounds on this quantity.The authors present simulation results confirming the intuitive behavior of DRLR, and also show (on real data) that the method (essentially) doesn't hurt.Overall, I thought this was a nice contribution to the growing line of work on distributional robustness, and is likely to spur follow-on work.The paper was very well written (although there were a few typos scattered throughout, e.g., "week" --> "weak" on L62).I did, however, have a few comments re: the empirical evaluation:- Firstly, it is not really that surprising (to me, anyway) that your accuracy is essentially the same as that of your competitors on the real data (since typically, you have to sacrifice something when you gain in robustness) -- did you consider reporting CVaR on the test set for all the methods (you might see a better gain here, and this might more precisely characterize the performance of your method)?- Secondly, could you provide some discussion on how tractable the optimization problems (7, 10a, 10b) actually are?What are the specs of the computer you are using to solve these problems?Do you have any timing results?- How sensitive is your method to the choice of norm (you seem to use the Linfinity, L1, and L2 norm in your experiments, which indicates to me that no one norm is suitable for all problems) and kappa?Is the guidance for selecting these hyper-parameters simply (some kind of) cross-validation (which, btw, adds significantly to the computational cost, but does hinge on my previous question re: timing results)? - Part of your (stated) motivation for pursuing robustness is sparsity in the design matrices -- I wonder if you could conduct simulations where you play with fat vs. skinny design matrices with different levels of sparsity.- I wonder if you could conduct some simulations to check the accuracy of Thm. 2, as the light-tailed assumption does seem restrictive to me (it is good that you included Fig. 3c in the paper).