summaries: This paper introduces a new method for training a classifier that simultaneously optimizes for a fairness criterion and robustness to data poisoning. The method is shown to increase measures of fairness and reduce inaccuracy on poisoned data relative to classifiers that only consider accuracy or fairness. Extensive results are shown for both synthetic and real benchmark data sets. This paper combines adversarial fair training with adversarial robust training. The basic idea is that a classifier is combined with two adversaries: one tries to predict the sensitive attribute $Z$ from the output of the classifier (essentially the approach by Edwards&Storkey 2015) and the other adversary tries to recognize if a label was predicted or is from a clean hold-out dataset. The latter is intended to harden the classifier against data-poisoning of the training set.
abstractText: We consider the problem of fair and robust model training in the presence of data poisoning. Ensuring fairness usually involves a tradeoff against accuracy, so if the data poisoning is mistakenly viewed as additional bias to be fixed, the accuracy will be sacrificed even more. We demonstrate that this phenomenon indeed holds for state-of-the-art model fairness techniques. We then propose FR-GAN, which holistically performs fair and robust model training using generative adversarial networks (GANs). We first use a generator that attempts to classify examples as accurately as possible. In addition, we deploy two discriminators: (1) a fairness discriminator that predicts the sensitive attribute from classification results and (2) a robustness discriminator that distinguishes examples and predictions from a clean validation set. Our framework respects all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity. Also, FR-GAN optimizes fairness without requiring the knowledge of prior statistics of the sensitive attributes. In our experiments, FR-GAN shows almost no decrease in fairness and accuracy in the presence of data poisoning unlike other state-of-the-art fairness methods, which are vulnerable. In addition, FR-GAN can be adjusted using parameters to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable.
references.authors: Alekh Agarwal Alina Beygelzimer Miroslav Dudı́k John Langford Hanna M. Wallach
references.title: A reductions approach to fair classification
references.venue: In ICML,
references.authors: J. Angwin J. Larson S. Mattu L. Kirchner
references.title: Machine bias: There’s software used across the country to predict future criminals
references.venue: And its biased against blacks.,
references.authors: Solon Barocas Selbst
references.title: Big Data’s Disparate Impact
references.venue: California Law Review,
references.authors: Rachel K.E. Bellamy Kuntal Dey Michael Hind Samuel C. Hoffman Stephanie Houde Kalapriya Kannan Pranay Lohia Jacquelyn Martino Sameep Mehta Aleksandra Mojsilovic Seema Nagar Karthikeyan Natesan Ramamurthy John T. Richards Diptikalyan Saha Prasanna Sattigeri Moninder Singh Kush R. Varshney Yunfeng Zhang
references.title: AI fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias
references.venue: CoRR, abs/1810.01943,
references.authors: Battista Biggio Blaine Nelson Pavel Laskov
references.title: Support vector machines under adversarial label noise
references.venue: In ACML, pp
references.authors: Battista Biggio Igino Corona Davide Maiorca Blaine Nelson Nedim Srndic Pavel Laskov Giorgio Giacinto Fabio Roli
references.title: Evasion attacks against machine learning at test time
references.venue: In ECML PKDD,
references.authors: Alexandra Chouldechova Aaron Roth
references.title: The frontiers of fairness in machine
references.venue: learning. CoRR,
references.authors: Andrew Cotter Maya R. Gupta Heinrich Jiang Nathan Srebro Karthik Sridharan Serena Wang Blake E. Woodworth Seungil You
references.title: Training well-generalizing classifiers for fairness metrics and other data-dependent constraints
references.authors: Andrew Cotter Heinrich Jiang Karthik Sridharan
references.title: Two-player games for efficient non-convex constrained optimization
references.venue: In ALT,
references.authors: Flávio du Pin Calmon Dennis Wei Bhanukiran Vinzamuri Karthikeyan Natesan Ramamurthy Kush R. Varshney
references.title: Optimized pre-processing for discrimination prevention
references.venue: In NeurIPS,
references.authors: Cynthia Dwork Moritz Hardt Toniann Pitassi Omer Reingold Richard Zemel
references.title: Fairness through awareness
references.venue: In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference,
references.authors: Michael Feldman Sorelle A. Friedler John Moeller Carlos Scheidegger Suresh Venkatasubramanian
references.title: Certifying and removing disparate impact
references.venue: In KDD, pp
references.authors: Benoı̂t Frénay Michel Verleysen
references.title: Classification in the presence of label noise: A survey
references.venue: IEEE Trans. Neural Netw. Learning Syst.,
references.authors: Sahaj Garg Vincent Perot Nicole Limtiaco Ankur Taly Ed Huai hsin Chi Alex Beutel
references.title: Counterfactual fairness in text classification through robustness
references.venue: AIES,
references.authors: Ian J. Goodfellow Jean Pouget-Abadie Mehdi Mirza Bing Xu David Warde-Farley Sherjil Ozair Aaron C. Courville Yoshua Bengio
references.title: Generative adversarial nets
references.venue: In NeurIPS,
references.authors: Ian J. Goodfellow Jonathon Shlens Christian Szegedy
references.title: Explaining and harnessing adversarial examples
references.venue: In ICLR,
references.authors: Alon Halevy Flip Korn Natalya F. Noy Christopher Olston Neoklis Polyzotis Sudip Roy Steven Euijong Whang
references.title: Goods: Organizing google’s
references.authors: Moritz Hardt Eric Price Nati Srebro
references.title: Equality of opportunity in supervised learning
references.venue: In NeurIPS,
references.authors: Dan Hendrycks Mantas Mazeika Duncan Wilson Kevin Gimpel
references.title: Using trusted data to train deep networks on labels corrupted by severe noise
references.venue: In NeurIPS,
references.authors: Heinrich Jiang Ofir Nachum
references.title: Identifying and correcting label bias in machine learning
references.authors: Christopher Jung Michael J. Kearns Seth Neel Aaron Roth Logan Stapleton Zhiwei Steven Wu
references.title: Eliciting and enforcing subjective individual
references.venue: fairness. CoRR,
references.authors: Faisal Kamiran Toon Calders
references.title: Data preprocessing techniques for classification without discrimination
references.venue: Knowl. Inf. Syst.,
references.authors: Faisal Kamiran Asim Karim Xiangliang Zhang
references.title: Decision theory for discrimination-aware classification
references.venue: In ICDM, pp
references.authors: Toshihiro Kamishima Shotaro Akaho Hideki Asoh Jun Sakuma
references.title: Fairness-aware classifier with prejudice remover regularizer
references.venue: In ECML PKDD, pp
references.authors: Michael Kearns Aaron Roth Saeed Sharifi-Malvajerdi
references.title: Average individual fairness: Algorithms, generalization and experiments, 2019
references.authors: Diederik P Kingma Jimmy Ba
references.title: Adam: A method for stochastic optimization
references.venue: arXiv preprint arXiv:1412.6980,
references.authors: Jon M. Kleinberg Sendhil Mullainathan Manish Raghavan
references.title: Inherent trade-offs in the fair determination of risk scores
references.venue: In ITCS,
references.authors: Pang Wei Koh Jacob Steinhardt Percy Liang
references.title: Stronger data poisoning attacks break data sanitization
references.venue: defenses. CoRR,
references.authors: Ron Kohavi
references.title: Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid
references.venue: In KDD, pp
references.authors: Jakub Konečnỳ H Brendan McMahan Felix X Yu Peter Richtárik Ananda Theertha Suresh Dave Bacon
references.title: Federated learning: Strategies for improving communication efficiency
references.venue: arXiv preprint arXiv:1610.05492,
references.authors: Alexey Kurakin Ian J. Goodfellow Samy Bengio
references.title: Adversarial machine learning at scale
references.venue: In ICLR,
references.authors: Yuncheng Li Jianchao Yang Yale Song Liangliang Cao Jiebo Luo Li-Jia Li
references.title: Learning from noisy labels with distillation
references.venue: In ICCV,
references.authors: Jianhua Lin
references.title: Divergence measures based on the Shannon entropy
references.venue: IEEE Transactions on Information theory,
references.authors: Nagarajan Natarajan Inderjit S. Dhillon Pradeep Ravikumar Ambuj Tewari
references.title: Learning with noisy labels
references.venue: In NeurIPS, pp
references.authors: Natasha Noy Matthew Burgess Dan Brickley
references.title: Google dataset search: Building a search engine for datasets in an open web ecosystem
references.venue: In 28th Web Conference (WebConf
references.authors: Adam Paszke Sam Gross Soumith Chintala Gregory Chanan Edward Yang Zachary DeVito Zeming Lin Alban Desmaison Luca Antiga Adam Lerer
references.title: Automatic differentiation in PyTorch
references.venue: In NIPS Autodiff Workshop,
references.authors: Andrea Paudice Luis Muñoz-González Emil C. Lupu
references.title: Label sanitization against label flipping poisoning attacks
references.venue: In ECML PKDD,
references.authors: Geoff Pleiss Manish Raghavan Felix Wu Jon M. Kleinberg Kilian Q. Weinberger
references.title: On fairness and calibration
references.venue: In NeurIPS,
references.authors: Mengye Ren Wenyuan Zeng Bin Yang Raquel Urtasun
references.title: Learning to reweight examples for robust deep learning
references.venue: In ICML,
references.authors: Andreas Veit Neil Alldrin Gal Chechik Ivan Krasin Abhinav Gupta Serge J. Belongie
references.title: Learning from noisy large-scale datasets with minimal supervision
references.venue: In CVPR,
references.authors: Suresh Venkatasubramanian
references.title: Algorithmic fairness: Measures, methods and representations
references.venue: In PODS,
references.authors: Sahil Verma Julia Rubin
references.title: Fairness definitions explained
references.venue: In FairWare@ICSE,
references.authors: Eric Wong J. Zico Kolter
references.title: Provable defenses against adversarial examples via the convex outer adversarial polytope
references.venue: In ICML,
references.authors: Tong Xiao Tian Xia Yi Yang Chang Huang Xiaogang Wang
references.title: Learning from massive noisy labeled data for image classification
references.venue: In CVPR,
references.authors: Mikhail Yurochkin Amanda Bower Yuekai Sun
references.title: Learning fair predictors with sensitive subspace robustness
references.venue: ArXiv, abs/1907.00020,
references.authors: Muhammad Bilal Zafar Isabel Valera Manuel Gomez-Rodriguez Krishna P. Gummadi
references.title: Fairness constraints: Mechanisms for fair classification
references.venue: In AISTATS,
references.authors: Richard S. Zemel Yu Wu Kevin Swersky Toniann Pitassi Cynthia Dwork
references.title: Learning fair representations
references.venue: In ICML, pp
references.authors: Brian Hu Zhang Blake Lemoine Margaret Mitchell
references.title: Mitigating unwanted biases with adversarial learning
references.venue: In AIES,
sections.heading: 1 INTRODUCTION
sections.text: As machine learning becomes widespread in the Software 2.0 era (Karpathy, 2017), there is an urgent need to address the issues of fairness and robustness. For sensitive applications like healthcare, finance, and self-driving cars, a trained model must not discriminate customers based on sensitive attributes including age, sex, or religion. In addition, as applications often rely on external data sets for their training data, the model training must be resilient against noisy or even adversarial data.
  Traditionally, model fairness research (Venkatasubramanian, 2019; Chouldechova & Roth, 2018; Verma & Rubin, 2018) has focused on developing metrics such as disparate impact (Feldman et al., 2015), equalized odds (Hardt et al., 2016), and equal opportunity (Hardt et al., 2016), which capture various notions of discrimination. More recently, there has been a surge in unfairness mitigation techniques (Bellamy et al., 2018), which improve the model fairness by either fixing the training data, training process, or trained model. Unfairness mitigation usually involves some tradeoff between the model’s accuracy and fairness. Most recently, generative adversarial networks (GANs) are being adapted to a fairness setting (Zhang et al., 2018). The architecture of GANs is suitable because it can be used to compete the training between two objectives: accuracy and fairness.
  Robust model training is also important and needs to be concurrently taken into consideration. There has been a proliferation of algorithms that make model training resilient against noisy and even adversarial data (Natarajan et al., 2013; Biggio et al., 2011; Frénay & Verleysen, 2014). The latter scenario may occur when the source of the training data is external, and we cannot control the data that is being added. For example, any public dataset on the Web may contain intentionallyfalse information. Moreover, one must have a robust training algorithm if the data set comes from diverse users’ inputs that cannot be fully trusted (Konečnỳ et al., 2016). Recently, data poisoning attacks have become increasingly sophisticated where it is becoming difficult to sanitize the data against all of them (Koh et al., 2018). In addition, dataset searching is becoming mainstream as demonstrated by Google Dataset Search (Goods) (Halevy et al., 2016) and its public version for searching scientific datasets (Noy et al., 2019). While data lakes within companies may consist of refined datasets, datasets in the public are easy to poison. And anyone can poison public datasets
  using attacks in the literature and share them. We thus believe that it is essential to address both bias and poisoning as a preventive measure.
  Solving model fairness without addressing data poisoning may lead to a worse tradeoff between accuracy and fairness. For example, consider a banking system that is giving out loans where there are two sensitive groups: men and women. According to disparate impact, the two groups must have similar ratios of positive predictions. A perfect disparate impact would have a value of 1. Suppose the clean training data has no bias and a perfect (or high) disparate impact score. Now suppose this data is poisoned where half of the positive labels of men are flipped to negative labels. If the model is fitted so as to make more negative predictions on men than women, then disparate impact would worsen. To address this problem, one may suggest using data sanitization techniques prior to the model training, but it is challenging to do so without any knowledge of the model. Even though many data sanitization methods are proposed, there have been newer attacks that can easily evade such defenses (Koh et al., 2018).
  Our main contribution is the development of a new framework called FR-GAN, which trains accurate models that are also fair and robust to poisoning. FR-GAN consists of a generator used for classification, a discriminator that distinguishes predictions from one sensitive group against others, and a second discriminator that distinguishes {examples, predictions} of the training data from {examples, labels} of a separate and clean validation set. The first discriminator ensures that the prediction ŷ is independent of the sensitive attribute z. We show that variations of this approach can be used to maximize any of the following prominent fairness measures: disparate impact, equalized odds, and equal opportunity. The second discriminator ensures that the model predictions on the training data are “consistent” with labels on clean data. We theoretically show that FR-GAN optimizes fairness without requiring the knowledge of prior statistics of sensitive attributes. In addition, the parameters of FR-GAN can be adjusted to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable. In the following sections, we present the related work, demonstrate the weaknesses of current fairness methods, and propose FR-GAN.
sections.heading: 2 RELATED WORK
sections.text: Model fairness The notion of discrimination has many definitions and usually comes from certain social goals that one wants to guarantee. Fairness research has been focused on defining fairness measures that capture a variety of notions of fairness (Verma & Rubin, 2018). There are largely two types of fairness measures: (1) group fairness (Barocas & Selbst, 2016; Hardt et al., 2016), which ensures similar statistics between two sensitive groups; (2) individual fairness (Dwork et al., 2012), which guarantees similar prediction results across nearby examples. Group fairness measures are widely studied in general applications, even though these cannot be simultaneously satisfied in certain circumstances (Kleinberg et al., 2017). In addition, there are also recent significant works that focus on individual fairness (Garg et al., 2018; Yurochkin et al., 2019; Kearns et al., 2019; Jung et al., 2019). Among the two types of fairness with different aspects, we focus on group fairness as our initial research. Recently, there has also been a surge of research on unfairness mitigation techniques (Bellamy et al., 2018). Depending on where a fix occurs, there are mainly three approaches: (1) pre-processing techniques (Kamiran & Calders, 2011; du Pin Calmon et al., 2017; Zemel et al., 2013; Feldman et al., 2015) that fix the training data; (2) in-processing techniques (Zafar et al., 2017; Jiang & Nachum, 2019; Zhang et al., 2018; Kamishima et al., 2012; Cotter et al., 2019; 2018; Agarwal et al., 2018) that address the issue during model training; and (3) post-processing techniques (Hardt et al., 2016; Pleiss et al., 2017; Kamiran et al., 2012) that manipulate predictions while maintaining the model.
  Among the three, the in-processing techniques have the advantages that one can work with any data and that there is more control on model training (Venkatasubramanian, 2019). FAIRNESS CONSTRAINTS (Zafar et al., 2017) incorporates a regularization term that reflects fairness constraints in the context of convex margin-based classifiers such as logistic regression and support vector machines (SVMs). The main hyperparameter is the λ term, which serves to balance the accuracy and fairness objectives. LABEL BIAS CORRECTION (Jiang & Nachum, 2019) assumes the existence of true yet possibly biased labels. A reweighting algorithm is proposed with theoretical guarantees that training on the resulting loss corresponds to training on the true unbiased labels, which yields a fair model. Here the hyperparameters include the number of training iterations. Finally, ADVERSARIAL
  DEBIASING (Zhang et al., 2018) trains a classifier so that an adversary cannot predict the sensitive attributes based on the output of the classifier. Hence, the classifier achieves fairness when its predictions do not have any correlation with sensitive attributes. This idea of adversarial training is similar to GANs (Goodfellow et al., 2014). ADVERSARIAL DEBIASING provides insights of using adversarial training to minimize mutual information between the prediction and sensitive attributes. In this paper, we strengthen the theoretical results of ADVERSARIAL DEBIASING using information theory, and this motivates us to provide systematic methodology for various fairness metrics (Section 4.1).
  As we demonstrate in Section 3, the existing techniques are not tailored for robust training, so are vulnerable to data poisoning attacks. In comparison, FR-GAN addresses both model fairness and robust training within the same model training process.
  Robust training There is a heavy literature on how to make the model training robust against noisy or even adversarial data (Natarajan et al., 2013; Biggio et al., 2011; Frénay & Verleysen, 2014; Kurakin et al., 2017). A major challenge is that there can be a wide range of data poisoning attacks that keep on evolving. One defense approach is to sanitize the training data and thus fix the root cause. A fundamental limitation is that we do not know in general what kind of poisoning will occur, so a universal defense against all possible attacks does not seem feasible as demonstrated by Koh et al. (2018). A more recent trend is to develop general defense algorithms for any attack during model training using meta learning (Veit et al., 2017; Li et al., 2017; Xiao et al., 2015; Hendrycks et al., 2018). Ren et al. (2018) propose a meta-learning framework that employs a clean validation set for the purpose of preventing the model training from being influenced by poisoned data. By using the validation loss as a meta objective, it intends to defend against any poisoning attack. Our FR-GAN framework is inspired by this approach, but employs a GAN-based model to support fair and robust training without using meta learning. In particular, the design of FR-GAN’s robustness discriminator is based on mutual-information-based theoretical insights (Section 4.2). Another line of research is defending against adversarial attacks during test time (Biggio et al., 2013; Goodfellow et al., 2015; Wong & Kolter, 2018). In comparison, our focus is on defending against data poisoning on the training data.
sections.heading: 3 VULNERABILITY OF FAIRNESS METHODS
sections.text: We perform experiments to demonstrate that state-of-the-art fairness methods are indeed vulnerable even to simple poisoning attacks. We generate a synthetic dataset as shown in Figure 1a (see the generation details in Section 5.1). There are two non-sensitive attributes a and b, which are reflected in the x-axis and y-axis, respectively. The examples are further divided into two classes based on the sensitive attribute z. To generate poisoned data, we measure accuracy degradation for a flipping of each z. We then choose the top-10% of the z values with the highest degradation and flip them. This approach is similar to the existing label flipping method (Paudice et al., 2018), except that the
  z value is being flipped instead of the label so that the model fairness is also affected. To make a validation set, we randomly select clean examples that amount to 10% of the entire training data.
  We use disparate impact (Feldman et al., 2015) as a fairness measure and evaluate one fairness method, called FAIRNESS CONSTRAINTS (Zafar et al., 2017). As this method involves a regularization factor λ that balances the accuracy and fairness objectives, we can obtain a tradeoff curve by adjusting its value. Figure 1b shows two accuracy-fairness tradeoff curves obtained with the clean and poisoned synthetic datasets. Notice that adding data poisoning clearly shifts the curve to the left, which means both accuracy and fairness decrease. This coincides with our intuition. The poisoning confuses the model so that there are more biased examples to fix, which in turn makes it overreact and thus sacrifice more on accuracy. In the supplementary, we show similar trends when the data is poisoned via label flipping. In Section 5, we will show how data poisoning affects LABEL BIAS CORRECTION (Jiang & Nachum, 2019) and ADVERSARIAL DEBIASING (Zhang et al., 2018).
sections.heading: 4 FR-GAN
sections.text: We now describe FR-GAN, which is shown in Figure 2. Unlike traditional GANs, the generator is a classifier that receives an example x ∈ X and returns a prediction ŷ. The other two discriminators optimize fairness and robustness as we describe in the following sections.
sections.heading: 4.1 FAIRNESS
sections.text: We denote by Dtr the training data set. Suppose Dtr has m examples {(x(i), z(i), y(i))}mi=1 where x(i) contains the non-sensitive attributes, z(i) contains the sensitive attributes, and y(i) is the label. Both the sensitive attribute and label can be multi-class, i.e., they can have one of multiple values. For notational simplicity, we assume there is one sensitive attribute, which can be viewed as a merged result of multiple sensitive attributes with a larger alphabet size. For illustrative purposes, we focus on disparate impact, leaving in the supplementary our formulation for equalized odds and equal opportunity.
  The first discriminator in FR-GAN distinguishes predictions w.r.t. one sensitive group from those in the others. Disparate impact intends the sensitive attribute to be independent of the model’s prediction, i.e.,
  I(Z; Ŷ ) = 0.
  We explain how FR-GAN can enforce the above constraint. Let PZ(z) be the distribution of Z where z ∈ Z and Z is the set of possible sensitive attribute values. Let Ŷ |Z = z ∼ PŶ |z(·) and Ŷ ∼ PŶ (·). Then PŶ (·) = ∑ z∈Z PZ(z)PŶ |z(·).
  We show that mutual information is equivalent to the following expression where the optimal discriminator D?z(ŷ) = PŶ ,Z(ŷ,z)
  PŶ (ŷ) and
  ∑ z∈Z D ? z(ŷ) = 1 ∀ŷ ∈ Y .
  Theorem 1. I(Z; Ŷ ) = maxDz(ŷ):∑z Dz(ŷ)=1∀ŷ∑z∈Z PZ(z)EPŶ |z [logDz(Ŷ )]+H(Z). While deferring the detailed proof to the supplemental materials, we provide a brief overview of the proof. As the optimization problem in the RHS is convex, we find the optimal dicriminator by solving the KKT conditions. We then show that the maximum value attained by the optimal discriminator is equal to the mutual information by using the properties of mutual information and the generalized Jensen-Shannon divergence (Lin, 1991).
  What is more involved than showing the above equality is designing a right optimization problem. One needs to carefully handcraft a plausible optimization problem so that its unique solution matches the desired quantity. Here, we design the above optimization problem via a ‘guess-&-check’ approach aided by the structural insights across the KL divergences that appear in an alternative expression of mutual information.
  We now discuss how to implement the above expression. Since we do not know PŶ |z(·) exactly, we compute the following empirical version:
  max Dz(ŷ): ∑ z Dz(ŷ)=1∀ŷ ∑ z∈Z PZ(z) ∑
  i:z(i)=z
  1
  mz logDz(ŷ
  (i)) +H(Z).
  Now for sufficiently large m, the number mz of examples with z(i) = z is approximately the same as PZ(z)m. Therefore, the above expression becomes:
  max Dz(ŷ): ∑ z Dz(ŷ)=1∀ŷ ∑ z∈Z ∑ i:z(i)=z 1 m logDz(ŷ (i)) +H(Z).
  Interestingly, this formulation is exactly the same as that in the original GAN (Goodfellow et al., 2014) when |Z| = 2. We also remark that our formulation does not require a prior knowledge on PZ(z).
sections.heading: 4.2 ROBUSTNESS
sections.text: We utilize a clean yet small validation set to ensure robust training. In particular, we add a second discriminator that distinguishes the training data with predictions {(x(i), z(i), ŷ(i))}mi=1 from the validation set {(x(i)val, z (i) val, y (i) val)} mval i=1 . Intuitively, if the classifier is confused by data poisoning in the training data, then its predictions will not be consistent with the labels of the clean data, and the discriminator would be able to detect that difference. While using a validation set is inspired by the meta learning approach by Ren et al. (2018), we take an adversarial learning approach. A key motivation behind this choice is that our GAN approach introduces a knob that controls the emphasis of robust training. We find that this knob enables FR-GAN to be more robust to the validation set size. See details in Section 5.1.
  Our formulation is as follows. Let V ∼ Bern(β) denote whether an example comes from training data (V = 1) or from the validation set (V = 0). We then want to ensure that V is independent of the training data and predictions, i.e., I(V ;X,Z, Ŷ ) = 0, which means the predictions on the training data are indistinguishable from labels of the validation set. This way we can mimic the clean data set while expecting an indirect sanitization effect. We optimize this independence using a formulation similar to that of fairness. Another possible formulation is to use the hard decision value from D(Ŷ ), say Ẑ, instead of Z. However, this discriminator can only be trained after the fairness discriminator is trained. Instead, we take an approach that allows us to train the two discriminators at the same time.
sections.heading: 4.3 ARCHITECTURE
sections.text: Figure 2 illustrates the architecture of FR-GAN. For the loss function in the generator, we employ the cross entropy loss:
  L1 = 1
  m m∑ i=1 −y(i) log ŷ(i) − (1− y(i)) log(1− ŷ(i)).
  As per Theorem 1, we set the loss function w.r.t. the fairness discriminator as:
  L2 = max D(·) ∑ z∈Z ∑ i:z(i)=z 1 m logDz(ŷ (i)) +H(Z)
  where D(·) := (D1(·), . . . , D|Z|(·)). The condition ∑ z∈Z D ? z(Ŷ ) = 1 can be enforced by adding a softmax layer to the discriminator.
  Finally, implementing I(V ;X,Z, Ŷ ) as advised by Theorem 1, we set the loss function w.r.t. the robustness discriminator as:
  L3 = max Dr(·) ∑ i:v(i)=0 1 m logDr(x(i), z(i), ŷ(i)) + ∑ i:v(i)=1 1 m log(1−Dr(x(i), z(i), ŷ(i))) +H(V ).
  The final objective function is to minimize the weighted sum of these value functions:
  min G(·) L1 + λ1L2 + λ2L3.
  Here λ1 and λ2 are tuning knobs that play roles to emphasize fair and robust training, respectively.
sections.heading: 5 EXPERIMENTS
sections.text: We provide experimental results for disparate impact, while leaving in the supplementary those for equalized odds and equal opportunity. All of our results are on a separate test set. More implementation details are also in the supplementary.
sections.heading: 5.1 SYNTHETIC DATA RESULTS
sections.text: For the synthetic data, we generate 2,000 examples with two non-sensitive attributes a and b, a sensitive attribute z, and a label y, using a method similar to the algorithm proposed by Zafar et al. (2017). Both z and y are binary and generated uniformly at random. For the four possible (z, y) combinations, we generate different normal distributions for a and b. Finally for each example, the a and b values are sampled as per the normal distribution associated with the (z, y) pair. For data poisoning, we flip the z values of examples so as to lower the model’s accuracy the most as described in Section 3.
  To make a fair comparison in the context of robust training, we also improve the existing fairness methods with data sanitization and compare them with FR-GAN. Since FR-GAN is one of the first works to address both fairness and robustness, there are few baselines to compare with. Hence, we employed one reasonable baseline, which first sanitizes the poisoned data using a well-known sanitization technique and then performs each fairness algorithm. There are several sanitization techniques surveyed in Koh et al. (2018), and we choose the loss defense method. Here examples that
  are not well fit by the model and result in a high loss are considered poisoned and are subsequently discarded. Loss defense is the only method that utilizes the trained model and thus has the best chance of defending poisoning attacks.
  Accuracy and fairness We compare FR-GAN with FAIRNESS CONSTRAINTS (Zafar et al., 2017), LABEL BIAS CORRECTION (Jiang & Nachum, 2019), ADVERSARIAL DEBIASING (Zhang et al., 2018), and a plain non-fairness method, logistic regression. We also compare with the meta learning approach by Ren et al. (2018), which focuses on robust training. For this comparison, we tailor our framework to a setting where λ1 = 0 (no fairness constraint), and name it R-GAN. For FR-GAN, we use a validation set that amounts to 10% of Dtr. When setting λ1 and λ2, we usually fix λ2 to some value and then adjust λ1. For the clean data, we apply proper hyperparameters so that the disparate impacts are similar (around 0.8) across all distinct methods. We do the same for the poisoned data. There is no such hyperparameter tuning for logistic regression and the meta learning by Ren et al. (2018), as the frameworks have no knob for adjusting fairness. We find that this leads to poor disparate impacts. For the three fairness methods, data poisoning aggravates performances in accuracy, fairness or both. For example, the disparate impact and accuracy of FAIRNESS CONSTRAINTS fall by 8.03% and 18.7%, respectively. On the other hand, the performance changes for FR-GAN are negligible: disparate impact increases by 2.09%, and accuracy decreases by 0.12%. Table 1 also shows that combining the fairness methods with loss defense (rows 6–8) does not always yield better accuracy and fairness. In fact, using the loss defense may lower the accuracy or fairness (e.g., LBC+LD has a disparate impact of 0.513 on poisoned data while LBC has 0.816). Perhaps this is because the data sanitization sometimes mistakenly discards clean data as well. The results demonstrate that even a data sanitization technique may not help much. In the supplementary, we also perform experiments using equalized odds and show that FR-GAN has similar benefits.
  We observe how accuracy trades off with fairness on clean and poisoned datasets. The results for FAIRNESS CONSTRAINTS are shown in Figure 1b. For LABEL BIAS CORRECTION, we employ the number of training iterations as a knob to trade accuracy off fairness. As shown in Figure 3a, the tradeoff curve shifts to the left, which demonstrates a clear tradeoff degradation. For ADVERSARIAL DEBIASING, we employ the α parameter (Zhang et al., 2018) analogous to λ1 as a knob to trade accuracy off fairness. We see in Figure 3b that the tradeoff curve again shifts to the left.
  Validation set size Figures 3c to 3e show how the validation set size affects the robustness of FRGAN. In particular, we compare the accuracy-fairness tradeoff of FR-GAN on clean data and that on poisoned data while varying the size of the validation set. When running on poisoned data, we fixed λ2 = 0.2 and varied λ1. We see that even 2% validation set (Figure 3d) is sufficient to maintain the accuracy and fairness obtained on the clean data. When using 0.5% (Figure 3e), the validation set is too small and has an adverse effect on the training. However, our framework includes a tuning knob λ2, so by decreasing λ2 down to 0.1, we could de-emphasize robust training, thereby avoiding
  the adverse effect (Figure 3f). This is in contrast to the meta learning approach, which suffers from a non-negligible performance degradation for a very small validation set. See details in the supplementary. The results show that only a holistic framework like FR-GAN can achieve both excellent model fairness and training robustness. In comparison, other methods tailored for only one of these objectives lose either accuracy, fairness or both.
  Ablation study (Without ‘R’) For a small value of λ2, we observe in Figure 3f that the accuracyfairness tradeoff curve shifts to the left, just like other fairness-only methods. To make the comparison clearer, we also plot them together in the supplementary. (Without ‘F’) When λ1 = 0, FR-GAN is expected to behave similarly to the other non-fair algorithms. The performance of FR-GAN with λ1 = 0, which we dub as R-GAN, is reported in Table 1. One can observe that R-GAN achieves a poor fairness performance just like the other non-fair algorithms.
sections.heading: 5.2 REAL DATA RESULTS
sections.text: We use the following benchmark datasets: ProPublica COMPAS (Angwin et al., 2016) and Adult Census (Kohavi, 1996). For sensitive attributes, we use SEX for both datasets. We do not employ the German Credit data set (Dua & Graff, 2017) as its highly biased towards positive labels. With such dataset, it is nearly impossible to observe any meaningful tradeoff.
  For data poisoning, we use the same method employed on synthetic data: flipping the z values so as to maximize the accuracy performance degradation. While there may be stronger attacks, this poisoning attack is effective enough to fool the three fairness methods. In the supplementary, we also evaluate FR-GAN against label flipping attacks and show similar results. We use the same constructions for the validation set and poisoning as described in Section 3.
  Accuracy and fairness See Table 2. As in Table 1, we apply proper hyperparameters so that disparate impacts are similar across all distinct methods, both for the clean and poisoned datasets. The results are similar to Table 1: logistic regression and meta learning exhibit poor disparate impacts; the three fairness methods have worse disparate impact and accuracy due to data poisoning; and FR-GAN again shows little degradation both in fairness and accuracy. Table 2 also shows that combining the fairness methods with loss defense (rows 6–8 and 15–17) does not always yield better accuracy and fairness and may even lower them, which is consistent to the results on synthetic data.
sections.heading: 6 DISCUSSION
sections.text: In this section, we discuss possible ways to extend FR-GAN.
  Subjective labels FR-GAN can be generalized to a setting where labels are subjective instead of being clean or poisoned. Here the dataset without the undesirable human biases becomes the clean validation set. In general, given two datasets with different distributions where one of them is desirable, FR-GAN can train robustly against the other distribution.
  Constructing real datasets We propose a general methodology for constructing a real dataset that has both bias and poisoning, as well as a “clean” validation set. Suppose that we are performing evaluation tasks (say loan decisions) where there is a high chance of human bias (i.e., poisoning) due to the high workload. We can construct a clean validation set by selecting a few tasks, assigning more evaluators, and taking a majority vote of the evaluations to minimize the bias. While employing more evaluators can be expensive, constructing a small validation set is sufficient for robust training, which makes this method practical.
  Generalizing to all attacks An interesting question is whether FR-GAN can defend against all poisoning attacks. If we know all the possible attacks, we can construct a training set containing these attacks. In the more challenging case where we do not know which attacks even exist, there seems to be a fundamental limitation in protecting against the attacks. Generalization is a universal problem in machine learning where a model trained on one dataset is not guaranteed to perform well in another dataset with a different distribution. Although the generalization is a critical issue to address, it is currently beyond the scope of this paper.
sections.heading: 7 CONCLUSION
sections.text: We proposed FR-GAN, which is a holistic framework that enables both unfairness mitigation and robust training. Our key contribution is proposing a novel GAN architecture that enjoys the synergistic effect of combining two approaches: (1) employing a fairness discriminator that distinguishes predictions w.r.t. one sensitive group from others and (2) employing a robust discriminator that distinguishes training data with predictions from a clean yet small validation set. We showed that existing fairness methods are vulnerable to data poisoning, even when combined with data sanitization techniques. In comparison, FR-GAN is robust to the poisoning and can be adjusted to maintain reasonable accuracy and fairness even if the validation set is too small or unavailable. We showed that FR-GAN is guaranteed to maximize fairness and supports all the prominent fairness measures: disparate impact, equalized odds, and equal opportunity.
sections.heading: A APPENDIX
sections.heading: A.1 PROOF FOR THEOREM 1
sections.text: Before we present the proof of the main theorem, we first recall our notation. Let PZ(z) be the distribution of Z where z ∈ Z and Z is the set of possible sensitive attribute values. Let Ŷ |Z = z ∼ PŶ |z(·) and Ŷ ∼ PŶ (·). Then PŶ (·) = ∑ z∈Z PZ(z)PŶ |z(·). Also, let Y ∼ PY (·).
  For convenience, let us repeat the statement of Theorem 1 here:
  I(Z; Ŷ ) = max Dz(ŷ): ∑ z Dz(ŷ)=1,∀ŷ ∑ z∈Z PZ(z)EPŶ |z [ logDz(Ŷ ) ] +H(Z).
  We now prove the theorem.
  Proof. Denote byD the collection of Dz(ŷ) for all possible values of z and ŷ, and by ν the collection of νŷ for all values of ŷ. We can construct the Lagrangian function as follows:
  L(D,ν) = ∑ z∈Z PZ(z)EPŶ |z [ logDz(Ŷ ) ] +H(Z) + ∑ ŷ∈Y νŷ ( 1− ∑ z∈Z Dz(ŷ) ) .
  We use the following KKT conditions:
  ∂L(D,ν) ∂Dz(ŷ) = PZ(z) PŶ |z(ŷ) D?z(ŷ) − ν?ŷ = 0, ∀(ŷ, z) ∈ Y × Z,
  1− ∑ z∈Z D?z(ŷ) = 0, ∀ŷ ∈ Y.
  Solving the two equations, we obtain ν?ŷ = PŶ (ŷ) for all ŷ. Thus,
  D?z(ŷ) = PZ(z)PŶ |z(ŷ)
  PŶ (ŷ) .
  Putting this to the above optimization,∑ z∈Z PZ(z)EPŶ |z [ log PZ(z)PŶ |z(Ŷ ) PŶ (Ŷ ) ] +H(Z)
  = ∑ z∈Z PZ(z)EPŶ |z
  [ log PZ(z)PŶ |z(Ŷ )
  PŶ (Ŷ )
  ] + ∑ z∈Z PZ(z) log 1 PZ(z)
  = ∑ z∈Z PZ(z)EPŶ |z
  [ log PŶ |z(Ŷ )
  PŶ (Ŷ ) ] = ∑ z∈Z PZ(z)DKL(PŶ |z‖PŶ )
  , JSPZ (PŶ |z1 , . . . , PŶ |z|Z|) = I(Z; Ŷ ).
  Here, the second last equality is due to the definition of the generalized JS divergence, and the last equality is due to its equivalence to the mutual information (Lin, 1991).
sections.heading: A.2 EXTENSIONS TO OTHER FAIRNESS MEASURES
sections.text: The following theorem relates the conditional mutual information I(Z; Ŷ |Y ) to the solution of an optimization problem. Theorem 2.
  I(Z; Ŷ |Y ) = max Dz|y(ŷ): ∑ z∈Z Dz|y(ŷ)=1;∀ŷ ∑ y∈Y ∑ z∈Z PY,Z(y, z)EPŶ |y,z [ logDz|y(Ŷ ) ] +H(Z|Y ).
  Recall that this conditional mutual information term can be used to capture equalized odds, which is another important fairness metric. We also note that the following theorem can be modified in a straightforward manner so that it can handle I(Z; Ŷ |Y = 1), which can be used to capture equal opportunity.
  We now prove the theorem.
  Proof. Denote by D the collection of Dz|y(ŷ) for all possible values of (z, ŷ, and y) and by ν the collection of νy,ŷ for all values of y and ŷ. We can construct the Lagrangian function as follows:
  L(D,ν) = ∑ y∈Y ∑ z∈Z PY,Z(y, z)EPŶ |y,z [ logDz|y(Ŷ ) ] +H(Z|Y ) + ∑ ŷ∈Y ∑ y∈Y νy,ŷ ( 1− ∑ z∈Z Dz|y(ŷ) ) .
  We use the following KKT conditions:
  ∂L(D,ν) ∂Dz|y(ŷ) = PY,Z(y, z) PŶ |y,z(ŷ) D?z|y(ŷ) − νy,ŷ = 0, ∀(ŷ, y, z) ∈ Y × Y × Z
  1− ∑ z∈Z D?z|y(ŷ) = 0, ∀(ŷ, y) ∈ Y × Y.
  Solving the two equations, we obtain ν?y,ŷ = PY,Ŷ (y, ŷ) for all (y, ŷ) ∈ Y × Y . Thus,
  D?z|y(ŷ) = PZ|y(z)PŶ |y,z(ŷ)
  PŶ |y(ŷ) ,∀y, ŷ ∈ Y × Y.
  Putting this to the above optimization,∑ y∈Y ∑ z∈Z PY,Z(y, z)EPŶ |y,z [ log PZ|y(z)PŶ |y,z(Ŷ ) PŶ |y(Ŷ ) ] +H(Z|Y )
  = ∑ y∈Y ∑ z∈Z PY,Z(y, z)EPŶ |y,z
  [ log PZ|y(z)PŶ |y,z(Ŷ )
  PŶ |y(Ŷ )
  ] + ∑ y∈Y ∑ z∈Z PY,Z(y, z) log 1 PZ|y(z)
  = ∑ y∈Y ∑ z∈Z PY,Z(y, z)EPŶ |y,z
  [ log PŶ |y,z(Ŷ )
  PŶ |y(Ŷ )
  ]
  = ∑ y∈Y ∑ z∈Z PY (y)PZ|y(z)EPŶ |y,z
  [ log PŶ |y,z(Ŷ )
  PŶ |y(Ŷ )
  ]
  = ∑ y∈Y PY (y) ∑ z∈Z PZ|y(z)EPŶ |y,z
  [ log PŶ |y,z(Ŷ )
  PŶ |y(Ŷ ) ] = ∑ y∈Y PY (y) ∑ z∈Z PZ|y(z)DKL(PŶ |y,z‖PŶ |y)
  , ∑ y∈Y PY (y) · JSPZ|y (PŶ |z1,y, . . . , PŶ |z|Z|,y)
  = ∑ y∈Y PY (y)I(Z; Ŷ |Y = y) = I(Z; Ŷ |Y ).
  The third last equality is due to the definition of the generalized JS divergence; the second last equality is due to its equivalence to the mutual information (Lin, 1991); and the last equality is due to the definition of conditional mutual information.
  We now discuss how to actually compute the mutual information. We compute the following empirical version using the example {(x(i), z(i), y(i))}mi=1.
  max Dz|y(ŷ): ∑ z∈Z Dz|y(ŷ)=1;∀ŷ ∑ y∈Y ∑ z∈Z PY,Z(y, z) ∑
  i:(y(i),z(i))=(y,z)
  1
  my,z logDz|y(ŷ
  (i)) +H(Z|Y ).
  Now for a sufficiently large value of m, my,z ≈ PY,Z(y, z)m. Therefore, the above expression is approximated as:
  max Dz|y(ŷ): ∑ z∈Z Dz|y(ŷ)=1;∀ŷ ∑ y∈Y ∑ z∈Z ∑ i:(y(i),z(i))=(y,z) 1 m logDz|y(ŷ (i)) +H(Z|Y ).
  Hence, we can set L2 (i.e., the loss w.r.t. the fairness discriminator) to the above expression. The rest of the objective function is the same. Figure 4 shows the resulting FR-GAN architecture.
sections.heading: A.3 ADDITIONAL EXPERIMENTS
sections.heading: A.3.1 SYNTHETIC DATA
sections.text: We continue our experiments from Sections 3 and 5.1.
  Label flipping for data poisoning Figure 5a to 5d show the experimental results when the data is poisoned via label flipping. We used the same synthetic data, as illustrated in Figure 1a, and flipped 7% of labels from the clean data to make a poisoned data. Figures 5a, 5b, and 5c show the accuracy-fairness tradeoff curves of FAIRNESS CONSTRAINTS, LABEL BIAS CORRECTION, and ADVERSARIAL DEBIASING, respectively. While the tradeoff curves in Figure 5a to 5c considerably shift to the left, Figure 5d shows that FR-GAN is noticeably robust against label flipping.
  Meta learning with different validation set sizes Table 3 shows the accuracy and fairness results for the meta learning method by Ren et al. (2018) (ML) for different validation set sizes. We observe a drastic decrease of accuracy when the validation set size is 0.1% of the training data.
  Ablation study for fairness Figure 6 shows the accuracy-fairness tradeoff curves for ADVERSARIAL DEBIASING and FR-GAN on the synthetic dataset. ADVERSARIAL DEBIASING tradeoff curves are from Figure 3b, and the FR-GAN curve comes from Figure 3f. We observe in Figure 6 that the accuracy-fairness tradeoff of FR-GAN using a small value of λ2 on the poisoned data is similar to the results of fairness-only methods.
sections.heading: A.3.2 REAL DATA
sections.text: We continue our experiments from Section 5.2.
  Label flipping for data poisoning Table 4 uses the same experimental setting as in Table 2, except that data poisoning is label flipping (not w.r.t. sensitive attributes). We observe similar trends as in Table 2. For the non-fairness methods, the fairness results are significantly worse than other methods. For the fairness-only methods, both fairness and accuracy tend to be high on the clean data, but at least one of them decreases significantly on the poisoned data. On the other hand, FR-GAN yields high fairness and accuracy on the clean data and is also the most resilient against poisoned data. The results emphasize our holistic technique that takes both fairness and accuracy into consideration.
sections.heading: A.3.3 FR-GAN USING OTHER FAIRNESS MEASURES
sections.text: As we showed in Appendix A.2, FR-GAN respects equalized odds and equal opportunity. Table 5 shows the experimental results on the synthetic and real datasets for equalized odds. We see that FR-GAN significantly improves equalized odds with reasonable accuracy. The results w.r.t. equal opportunity are similar and thus not shown here.
sections.heading: A.4 TRAINING METHODOLOGY
sections.text: We implement all algorithms in PyTorch (Paszke et al., 2017), and all experiments are performed on a server with Intel i7-6850 CPUs.
  The generator G is a neural network with zero or one hidden layer. The discriminator Df is a single layer neural network, and the discriminatorDr is a neural network with one hidden layer. We used 8 or 16 nodes in the hidden layers. We set an Adam optimizer (Kingma & Ba, 2014) for the generator,
  and a stochastic gradient descent (SGD) optimizer for each discriminator. We empirically observe that one can stabilize the training procedure by freezing the parameters of the fairness discriminator Df for the initial phase of training. Thus, we choose to freeze the parameters of the fairness discriminator Df for the first few epochs until the generator achieves a certain accuracy. We use the generator/discriminator update ratio of 1:1 for the first 300 epochs and use the update ratio of 1:3 (or 1:5) for the rest of training.
  Also, we use the following details for choosing the values of λ1 and λ2. For clean data, we set λ2 as a small value (e.g., 0.01) and vary λ1 from 0 to 0.95. For poisoned data, we set λ2 as 0.2, 0.3, or 0.4, and vary λ1 from 0 to 0.95 − λ2. Given the values of λ1 and λ2, we also normalize L1 (the generator loss) by multiplying it with (1− λ1 − λ2).
sections.heading: A.5 ADDITIONAL EXPERIMENTS ON ADULT DATASET
sections.text: We compare FR-GAN and AD using the AdultCensus dataset and the same poisoning in Section 3. For a fair comparison, we use loss defense (LD) as data sanitization prior to running AD. We generate confusion matrices both for disparate impact (DI) and equalized odds (EO). Overall, FRGAN performs better than AD because of its robustness discriminator. The robustness discriminator successfully ignores the poisoned distribution in the training data, and FR-GAN’s TPR is higher than AD with sanitization. Table 6 shows confusion matrices as results of disparate-impact-focused experiment. The results are reported when AD achieves (Acc, DI) = (0.728, 0.821), and FR-GAN achieves (Acc, DI) = (0.827, 0.866). In addition, Table 7 represents confusion matrices as results of equalized-odds-focused experiment. The results are reported when AD achieves (Acc, EO when y = 1, EO when y = 0) = (0.788, 0.844, 0.750), and FR-GAN achieves (Acc, EO when y = 1, EO when y = 0) = (0.838, 0.824, 0.773).