Quantifying, enforcing and implementing fairness emerged as a major topic in machine learning. We investigate these questions in the context of deep learning. Our main algorithmic and theoretical tool is the computational estimation of similarities between probability, "à la Wasserstein", using adversarial networks. This idea is flexible enough to investigate different fairness constrained learning tasks, which we model by specifying properties of the underlying data generative process. The first setting considers bias in the generative model which should be filtered out. The second model is related to the presence of nuisance variables in the observations producing an unwanted bias for the learning task. For both models, we devise a learning algorithm based on approximation of Wasserstein distances using adversarial networks. We provide formal arguments describing the fairness enforcing properties of these algorithm in relation with the underlying fairness generative processes. Finally we perform experiments, both on synthetic and real world data, to demonstrate empirically the superiority of our approach compared to state of the art fairness algorithms as well as concurrent GAN type adversarial architectures based on Jensen divergence.
This paper proposes a variant of adversarial learning to achieve some of the popular group fairness definitions. The main novelty is the idea of minimizing Wasserstein distance between the conditional distributions of classifier predictions given different values of the protected attribute.
My main concern is the approximation of a simple 1d Wasserstein distance with a neural network. Wasserstein distance between two discrete distributions in 1d can be computed in closed form (simple function of order statistics). That is, eq. (1) is simple to evaluate for two empirical distributions. There is no need to use a neural network for approximation, and even if authors choose to do so, some discussion on how well it approximates actual Wasserstein distance is needed. I think the proposed algorithm could be more interesting if authors can work out the optimization problem with the actual Wasserstein distance.
On the theoretical/motivation side, it is not enough to say that demographic parity is achieved when the corresponding Wasserstein distance is 0. What is needed is that demographic parity difference is bounded from above by the corresponding Wasserstein distance (I don't know if it is true or not, but would like to know). Then minimizing Wasserstein distance to achieve demographic parity could be justified.
Finally, the paper is quite poorly written. The description of fairness in the introduction is very vague. Authors essentially describe demographic parity as fairness, while it is simply one of the several definitions of group fairness. There is also individual fairness (the paper by Dwork et al. is cited, but not properly discussed) and prior work emphasizing certain deficiencies of group fairness [1] along with several recent papers studying individual fairness [2,3], some also utilizing Wasserstein distance [4].
Authors also provided incorrect definition of disparate impact. Equation in the bottom of page 2 corresponds to statistical parity difference, while disparate impact is the ratio.
"Equality of opportunity" on the top of page 3 seems to be a typo
"the mathematical properties of the disparate impact measure are not favorable, in particular it lacks robustness and smoothness features which would be necessary to blend algorithmic practice and mathematical theory" - I don't think this claim makes sense. There are many prior works studying disparate impact and proposing algorithms to achieve it, e.g. the cited work of Feldman et al. Authors should be more specific regarding what mathematical properties they consider not favorable.
There are a lot of typos and grammatical mistakes, e.g.
in the 1st paragraph of section 2.2, the sentence "Hence the aim in this case is to” is unfinished.
in the 1st paragraph of section 3, the first sentence seems to be unfinished.
[1] Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores.
[2] Kearns, M., Roth, A., & Sharifi-Malvajerdi, S. (2019). Average Individual Fairness: Algorithms, Generalization and Experiments.
[3] Jung, C., Kearns, M., Neel, S., Roth, A., Stapleton, L., & Wu, Z. S. (2019). Eliciting and Enforcing Subjective Individual Fairness.
[4] Yurochkin, M., Bower, A., & Sun, Y. (2019). Learning fair predictors with Sensitive Subspace Robustness.