Empirical Risk Minimization Under Fairness Constraints
We address the problem of algorithmic fairness: ensuring that sensitive information does not unfairly influence the outcome of a classifier. We present an approach based on empirical risk minimization, which incorporates a fairness constraint into the learning problem. It encourages the conditional risk of the learned classifier to be approximately constant with respect to the sensitive variable. We derive both risk and fairness bounds that support the statistical consistency of our methodology. We specify our approach to kernel methods and observe that the fairness requirement implies an orthogonality constraint which can be easily added to these methods. We further observe that for linear models the constraint translates into a simple data preprocessing step. Experiments indicate that the method is empirically effective and performs favorably against state-of-the-art approaches.
Overview: This paper presents an approach to fairness in statistical learning based on surrogate losses. They work in a generalization of the "equality of opportunity" (EO) framework of Hardt, Price, and Srebro in which the learner performs an ERM problem subject to a constraint that (perhaps only approximately) enforces equality of opportunity. The EO framework is relaxed so that we require equality of some abstract loss function across protected attributes rather than equality of probability that the classifier outputs a specific class. The EO constraint is also relaxed so that it only needs to hold approximately.   After introducing this framework, the authors show relax zero-one losses appearing in the objective or the EO constraint to surrogate losses. Specifically, they show how to bound approximation error when the zero-one loss in the ERM objective is replaced with the hinge loss and the zero-one loss in the EO constraint is replaced with the linear loss. For the special case where the function class is an RKHS they show to directly solve the fair ERM problem under this relaxation.  Finally, the authors conduct extensive experiments against comparable fair learning algorithms and find that their surrogate loss approach is competitive for linear classes.   Originality and Significance: On the mathematical side, I was somewhat disappointed with the limited scope of the paper's theory results relative to the breadth suggested by the title. In particular, the results appearing in the main body, theorem 1 and proposition 2, use mostly standard calculations from learning theory and in particular surrogate losses. It is indeed unfortunate that proposition 2 is limited to the special case of hinge loss and linear loss, and does not give a general framework. My conclusion here is that in terms of analysis tools there are not many new results or tricks that a broader audience might hope to take away.  I am willing to accept however that this paper makes a nice conceptual contribution, namely for the special case of their general problem (hinge/linear loss, linear function class) that is the focus of page 5 onward, the experimental evaluation the authors perform suggests that this method indeed outperforms that of comparable algorithms from Hardt et al. and Zafar et al. So I believe the method has practical merit and could be a useful baseline/benchmark for future development in this area. There is one caveat which is that all the algorithms compared against use the same linear representation. It would not suprise me if using one of the algorithms from [18-26] that direct solves the nonconvex problem for a more expressive function class (eg decision forests) can obtain higher accuracy and EO simultaneously, so this seems like an important benchmark.  To make the theory results stronger in the future it would be nice to give comparison results in the vein of proposition 2 for more general loss functions, as well as to try to establish proper classification calibration results in the vein of Bartlett, Jordan, and McAullife.  Quality and Clarity: I found the paper easy to read and follow overall. A couple minor notes on the writing: * People not already familiar with surrogate losses may find the discussion on page 4 unclear. * Try to define all the named losses (\ell_h, \ell_c, \ell_l) in the same place to avoid making people search for their definitions. * Please try to find a way to highlight the conclusions of Table 1 visually, eg highlight highest accuracy and EO for each dataset.  I appreciate the fairly extensive discussion of the experimental setup in the main paper and appendix.