Fairness with Wasserstein Adversarial Networks
Quantifying, enforcing and implementing fairness emerged as a major topic in machine learning. We investigate these questions in the context of deep learning. Our main algorithmic and theoretical tool is the computational estimation of similarities between probability, ```a la Wasserstein**, using adversarial networks. This idea is flexible enough to investigate different fairness constrained learning tasks, which we model by specifying properties of the underlying data generative process. The first setting considers bias in the generative model which should be filtered out. The second model is related to the presence of nuisance variables in the observations producing an unwanted bias for the learning task. For both models, we devise a learning algorithm based on approximation of Wasserstein distances using adversarial networks. We provide formal arguments describing the fairness enforcing properties of these algorithm in relation with the underlying fairness generative processes. Finally we perform experiments, both on synthetic and real world data, to demonstrate empirically the superiority of our approach compared to state of the art fairness algorithms as well as concurrent GAN type adversarial architectures based on Jensen divergence.
The authors propose a method for adding an approximate disparate impact loss to a classification objective, and show that optimizing a classifier for this loss leads to *fairer* predictions with little or no accuracy loss. The authors first formulate two notions of fairness in terms of earth mover distance between the distribution of scores conditioned on either value of a protected variable. They then show that the dual formulation of the earth mover distance can be approximated (specifically, lower-bounded) by optimizing the parameters of a neural network under spectral norm constraints. This leads to a min-max global optimization scheme to learn a fair classifier. Strengths: The proposed method does better on the considered fairness metric than a GAN model with similar accuracy. Weaknesses: The paper is difficult to read, glosses over some important details, and contains some inaccuracies. -- Clarity: --- The authors need to better describe the assumptions (or lack thereof) made on the joint distribution of X, S, and Y. --- Measures such as the quantiles or probability laws need to be formally defined before they are used in definitions. --- The mathcal{L} notation is overloaded (it is used for probability laws, conditional and unconditional, as well as marginals, with mathcal{L}_1 referring to both!), leading to potential confusion. --- The domain of X and Y in equation (2) is not defined anywhere, neither is the distance. --- Similar lack of consistency with the use of F / mathcal{F} / hat{f}, without any explicit parameterization --- Figure 2 needs to be in Section 4, and Table 1 needs to be trimmed to size -- Overlooked problems: --- In the dual formulation, the optimization is done over a sub-set of Lipschitz function, hence approximation of mathcal{W} is a lower bound at every step. Minimizing a lower bound on a loss can be justified, but requires more discussion --- The trade-off inherent in the choice of n_w in algorithm 1 needs to be further discussed, especially in the case of large datasets where a full epoch of SGD in the inner loop of the optimization process is impractical -- Inaccuracies: The graphical models in Figures 1 and 2 and conditional independences written in the text are not consistent: --- In Figure 1, X is NOT independent of S given Y (neither is Y*) (see: V structures in a directed graphical model) --- In Figure 2, X* is independent of S regardless of conditioning on Y Considering all of the above issues, the paper is not currently ready for publication
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Fairness with Wasserstein Adversarial Networks
Quantifying, enforcing and implementing fairness emerged as a major topic in machine learning. We investigate these questions in the context of deep learning. Our main algorithmic and theoretical tool is the computational estimation of similarities between probability, ```a la Wasserstein**, using adversarial networks. This idea is flexible enough to investigate different fairness constrained learning tasks, which we model by specifying properties of the underlying data generative process. The first setting considers bias in the generative model which should be filtered out. The second model is related to the presence of nuisance variables in the observations producing an unwanted bias for the learning task. For both models, we devise a learning algorithm based on approximation of Wasserstein distances using adversarial networks. We provide formal arguments describing the fairness enforcing properties of these algorithm in relation with the underlying fairness generative processes. Finally we perform experiments, both on synthetic and real world data, to demonstrate empirically the superiority of our approach compared to state of the art fairness algorithms as well as concurrent GAN type adversarial architectures based on Jensen divergence.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors propose a method for adding an approximate disparate impact loss to a classification objective, and show that optimizing a classifier for this loss leads to *fairer* predictions with little or no accuracy loss. The authors first formulate two notions of fairness in terms of earth mover distance between the distribution of scores conditioned on either value of a protected variable. They then show that the dual formulation of the earth mover distance can be approximated (specifically, lower-bounded) by optimizing the parameters of a neural network under spectral norm constraints. This leads to a min-max global optimization scheme to learn a fair classifier. Strengths: The proposed method does better on the considered fairness metric than a GAN model with similar accuracy. Weaknesses: The paper is difficult to read, glosses over some important details, and contains some inaccuracies. -- Clarity: --- The authors need to better describe the assumptions (or lack thereof) made on the joint distribution of X, S, and Y. --- Measures such as the quantiles or probability laws need to be formally defined before they are used in definitions. --- The mathcal{L} notation is overloaded (it is used for probability laws, conditional and unconditional, as well as marginals, with mathcal{L}_1 referring to both!), leading to potential confusion. --- The domain of X and Y in equation (2) is not defined anywhere, neither is the distance. --- Similar lack of consistency with the use of F / mathcal{F} / hat{f}, without any explicit parameterization --- Figure 2 needs to be in Section 4, and Table 1 needs to be trimmed to size -- Overlooked problems: --- In the dual formulation, the optimization is done over a sub-set of Lipschitz function, hence approximation of mathcal{W} is a lower bound at every step. Minimizing a lower bound on a loss can be justified, but requires more discussion --- The trade-off inherent in the choice of n_w in algorithm 1 needs to be further discussed, especially in the case of large datasets where a full epoch of SGD in the inner loop of the optimization process is impractical -- Inaccuracies: The graphical models in Figures 1 and 2 and conditional independences written in the text are not consistent: --- In Figure 1, X is NOT independent of S given Y (neither is Y*) (see: V structures in a directed graphical model) --- In Figure 2, X* is independent of S regardless of conditioning on Y Considering all of the above issues, the paper is not currently ready for publication
assistant
The authors propose a method for adding an approximate disparate impact loss to a classification objective, and show that optimizing a classifier for this loss leads to *fairer* predictions with little or no accuracy loss. The authors first formulate two notions of fairness in terms of earth mover distance between the distribution of scores conditioned on either value of a protected variable. They then show that the dual formulation of the earth mover distance can be approximated (specifically, lower-bounded) by optimizing the parameters of a neural network under spectral norm constraints. This leads to a min-max global optimization scheme to learn a fair classifier. Strengths: The proposed method does better on the considered fairness metric than a GAN model with similar accuracy. Weaknesses: The paper is difficult to read, glosses over some important details, and contains some inaccuracies. -- Clarity: --- The authors need to better describe the assumptions (or lack thereof) made on the joint distribution of X, S, and Y. --- Measures such as the quantiles or probability laws need to be formally defined before they are used in definitions. --- The mathcal{L} notation is overloaded (it is used for probability laws, conditional and unconditional, as well as marginals, with mathcal{L}_1 referring to both! ), leading to potential confusion. --- The domain of X and Y in equation (2) is not defined anywhere, neither is the distance. --- Similar lack of consistency with the use of F / mathcal{F} / hat{f}, without any explicit parameterization --- Figure 2 needs to be in Section 4, and Table 1 needs to be trimmed to size -- Overlooked problems: --- In the dual formulation, the optimization is done over a sub-set of Lipschitz function, hence approximation of mathcal{W} is a lower bound at every step. Minimizing a lower bound on a loss can be justified, but requires more discussion --- The trade-off inherent in the choice of n_w in algorithm 1 needs to be further discussed, especially in the case of large datasets where a full epoch of SGD in the inner loop of the optimization process is impractical -- Inaccuracies: The graphical models in Figures 1 and 2 and conditional independences written in the text are not consistent: --- In Figure 1, X is NOT independent of S given Y (neither is Y*) (see: V structures in a directed graphical model) --- In Figure 2, X* is independent of S regardless of conditioning on Y Considering all of the above issues, the paper is not currently ready for publication