Why Is My Classifier Discriminatory?
Recent attempts to achieve fairness in predictive models focus on the balance between fairness and accuracy. In sensitive applications such as healthcare or criminal justice, this trade-off is often undesirable as any increase in prediction error could have devastating consequences. In this work, we argue that the fairness of predictions should be evaluated in context of the data, and that unfairness induced by inadequate samples sizes or unmeasured predictive variables should be addressed through data collection, rather than by constraining the model. We decompose cost-based metrics of discrimination into bias, variance, and noise, and propose actions aimed at estimating and reducing each term. Finally, we perform case-studies on prediction of income, mortality, and review ratings, confirming the value of this analysis. We find that data collection is often a means to reduce discrimination without sacrificing accuracy.
Summary/Contribution This paper shows that disparities in many fairness measures can be decomposed into “bias” (loss caused by deviating from the best feasible classifier), “variance” (loss caused by randomness in the dataset), and “noise” (the loss of the best feasible classifier). In doing so it illuminates some important issues that need to be considered when attempting to improve group fairness measures. For example, many people suspect that high error rates (either FPR or FNR) for a minority group is at least partially due to a lack of minority data. This work shows that these error rates can persist even with plentiful data (ie when the contribution of “variance” to the error rate differences is zero).   Crucially, this paper recognizes that “existing approaches for reducing discrimination induced by prediction errors may be unethical or impractical to apply in settings where predictive accuracy is critical, such as in healthcare or criminal justice,” and proposes concrete means to improve algorithms without bearing this unacceptable cost.  Weaknesses: The biggest weakness of this paper is its use of the word “discrimination” to refer to differences in cost functions between groups. The authors provide no justification for why differences in (for example) zero-one loss should be termed “discriminatory,” evoking injustice and animus in the reader’s mind. A classifier that tries to predict who will become an NFL quarterback will have a worse zero-one loss for men than women (since no woman has ever played in the NFL), but this is absolutely not discrimination. I’d give this paper an "overall score" of 8 if they used a more value-neutral term (such as “disparities” or “differences”) instead of “discrimination”.  The authors also suggest a method for determining where to expend effort to produce better models, which looks at subgroups where error rates differ by protected characteristics. For example, they find that their model seems to make far fewer errors on Hispanic cancer patients than cancer patients of other races. I’d be interested to see whether this effect was due to differences in base rates within the subgroups. One group might be easier to classify simply because their base rate is closer to zero or one, but this doesn’t necessarily suggest the existence of extra features to reduce noise (and therefore the accuracy gap).   Clarity: This paper is well written and the math is clear.  Originality: This paper is clearly original, although other papers have hinted at similar high-level results.  Significance: This paper contains some valuable insights that could be readily applied to existing machine learning workflows, a rare achievement for FATML papers. However, I suspect the practical impact will be small since many problems have lots of data (so bias and variance are small) and no good way to get additional predictive features (so noise can't be reduced).  Response to author rebuttal: I was disappointed to see the authors push back about the use of "discrimination" to describe differences in cost functions. "However, in the NFL example, if differences in error rates were related to race, there might be cause for concern--depending on how the classifier is used. Drawing on terminology from prior work, we use “discrimination” as we consider applications in which decisions based on error-biased classifiers may be considered unfair, such as healthcare, employment and criminal justice—similar to how a human making consistent race-dependent errors could also be considered discriminatory." This is all true, differences in error rates *might* be discriminatory, but they also might not be. This is why we need to be extremely precise with our language. Previous papers have been careless with the word "discriminatory", but this does not mean this (otherwise careful and important) paper should follow their example. The title of the paper does not give me confidence that the discussion in the revised paper about when, exactly, differences in cost functions are discriminatory will be sufficiently nuanced.