Satisfying Real-world Goals with Dataset Constraints
The goal of minimizing misclassification error on a training set is often just one of several real-world goals that might be defined on different datasets. For example, one may require a classifier to also make positive predictions at some specified rate for some subpopulation (fairness), or to achieve a specified empirical recall. Other real-world goals include reducing churn with respect to a previously deployed model, or stabilizing online training. In this paper we propose handling multiple goals on multiple datasets by training with dataset constraints, using the ramp penalty to accurately quantify costs, and present an efficient algorithm to approximately optimize the resulting non-convex constrained optimization problem. Experiments on both benchmark and real-world industry datasets demonstrate the effectiveness of our approach.
The paper proposes a model and algorithm to solve linear classification problems on datasets with some common side-constraints that are often ignored.
I really like the basic premise of the paper. I am no expert on deployed systems, but it seems like these authors know about real-world systems (and have access to some proprietary real datasets) and have thought carefully about actual issues that come up when making a classifier.  The paper proposes (1) a new mindset, (2) a new model, and (3) an algorithm to solve the model. At times, I think any one of these components is a bit thin, and it’s not always clear why certain decisions (especially for the algorithm) were made — in particular, the numerical examples are provided to validate the new mindset but do not justify the algorithmic decisions. But overall, I think this is sufficiently novel that I am supportive.  A few specific points:  - I found the descriptions of the issues on page 1 and 2 (e.g., the churn example) quite interesting  - Page 3 and further in the paper, referring to “Problem 2” is a bit confusing when it is only labeled by an equation number (most authors define a problem environment, like a theorem). I also didn’t see r_p and r_n defined (line 103) — are these s_p and s_n, or a (un)normalized version?  - The randomized prediction (line 105) didn’t seem like it was fully explored. Does this affect training or only the final prediction? After vaguely introducing this concept, I felt like it was ignored until the numerical experiment plots, and I do not understand exactly what you are doing.  - Fig 1 was designed to make the convex bounds intuitive, but this means that for z >= 1/2, at the next step, there is no actual dependence on this component of z? With the hinge loss, at least there is still dependence. Intuitively this makes your scheme sound like a bad idea.    — Furthermore, discussing Davenport (line 178-180), you say that ramp-loss is similar or slightly better in accuracy than hinge-loss, but much faster. This doesn’t make sense to me, since to solve the ramp-loss you must solve a sequence of hinge-loss problems, so how can it be faster? Could you elaborate?  - Section 2.2 uses SVM solvers inside a cutting-plane routine. This may be a good idea, but you could describe why you didn’t pursue alternatives. Everything is piecewise linear, except the regularization which is quadratic, so you could solve everything with an off-the-shelf QP solver, which seems much simpler. Perhaps you prefer existing SVM solvers since you know good ones, and many of them naturally take in datasets in specified formats (such as libsvm format)?  A bit of discussion on your design decisions would be useful.