\section{Appendix}

\subsection{Recap of Recent Calibration Error Literature} \label{App: CE Metrics}
Calibration error metrics measure the calibration of the model (i.e., how closely the confidence of the model corresponds to the accuracy). While this appears to be a straightforward statistic to estimate, there are several ways to define and measure this notion and recent literature has found that not all methods are made equal. These nuances are important to clarify for the measurement of predictive parity as we have defined it based on the worst-group calibration error (refer to Section \ref{subsection: PP and Cal} for our reasoning). We now dive into these details by recapping research into calibration error metrics and then justifying our choice of using the Expected Cumulative Calibration Error (ECCE).

We denote the calibration of a measurement function $f$ as $CE(f)$. One of the most widely used measures of calibration error is the Expected Calibration Error (ECE) as proposed in \cite{naeini2015bbq} and widely popularized in \cite{guoCalibration}. ECE aims to capture the discrepancy between confidence and accuracy. Intuitively, ECE is computed by first splitting up the model score range into $M$ evenly spaced bins (e.g., [0, 0.1), [0.1, 0.2), etc.), then computing the expectation in outcome $Y_{m}$ in each bin (known as the bin accuracy $acc(B_{m})$) and the expected score $s$ (also known as the bin confidence $conf(B_{m})$) and taking the difference of the two terms. In terms of actual formulas, this is captured as:
\begin{equation}
    ECE = \sum_{m=1}^{M} \frac{\lvert B_{m} \rvert}{n} \lvert acc(B_{m})-conf(B_{m}) \rvert \label{eq: ECE}
\end{equation}

Where $M$ corresponds to an a-priori selected number of bins to evaluate and $n$ is the total number of samples.  However, several recent works have observed theoretical and empirical deficiencies with this metric \cite{Kumar2019, roelofs2022mitigating, Gruber2022, BlasiokCalibration, arrieta2022metrics}. To summarize these criticisms, ECE is a statistically biased (under-estimator) measure of true calibration error due to the need to parameterize the measurement with a a-priori fixed number of $M$ bins. The bias is worsened by ECE's sensitivity to sample size \cite{roelofs2022mitigating, Gruber2022}. The aforementioned works all propose alternative CE measurements that demonstrate better empirical performance \cite{roelofs2022mitigating} or have stricter theoretical requirements and properties such having a its distance to some true calibration error (TCE) be polynomially bounded (e.g., $CE(f) \leq \mathcal{O}(TCE^{s})$ for some constant $s$ \cite{BlasiokCalibration}) or have asymptotic convergence properties to the true calibration error \cite{arrieta2022metrics}. After evaluating the recent landscape of CE metrics, we have chosen to actively avoid using ECE and instead focus on three such metrics, described below in order of preference with accompanying justification. We recommend \cite{Gruber2022} and \cite{BlasiokCalibration} for a thorough survey of the CE landscape.

\begin{itemize}
  \item \textbf{Expected Cumulative Calibration Error} (ECCE): Proposed in \cite{arrieta2022metrics}, this metric offers a theoretically-backed, parameter-free, and scalable CE metric that enables hypothesis testing. The crux of this metric is that instead of using notions related to binning, they use the cumulative calibration error which presents asymptotic properties of converging to zero for a perfectly calibrated set of scores and being bounded away from zero for uncalibrated scores. Across our experiments, we rely on this CE the most to draw conclusions as it is has both theoretical properties and empirically works well. \footnote{Based on experiments following the bias-by-construction framework as presented in \cite{roelofs2022mitigating}}
  \item \textbf{Minimum Sweep Calibration Error} (MSCE): Proposed in \cite{roelofs2022mitigating}, this metric is a modification of the equal-mass binning version of the original ECE~\eqref{eq: ECE} where the number of bins is dynamically selected. The authors also validate its empirical effectiveness over ECE through the proposed bias-by-construction framework, where the $CE(f)=E_{D}[CE(f)]-TCE$ where $D$ is the distribution of scores and labels and $TCE$ is the true calibration error, which has a parametric form and thus can be numerically approximated. We favor this CE due to its empirical effectiveness and empirical alignment with the traditional ECE metric.
  \item \textbf{Kernel Calibration Error} (KCE): Proposed in \cite{kumar2018trainable} and further analyzed in \cite{BlasiokCalibration}, this metric is a reproducing kernel Hilbert space (RKHS) based measurement of calibration. Originally designed to be a differentiable and consistent estimator of ECE, Błasiok et al. show that it also carries theoretical properties of being a polynomial upper and lower bound to a notion of true calibration error \cite{BlasiokCalibration}. Although this CE has strong theoretical properties, it is expensive to compute without sampling as it requires $\mathcal{O}(n^{2})$ computations. As noted in \cite{BlasiokCalibration}, a practical implementation can bypass the $n^{2}$ scaling by drawing $M$ samples instead and relying on the error shrinkage rate of $\mathcal{O}(\frac{1}{\sqrt{M}})$. However, this results in another practical problem in that the measurement is stochastic. For these reasons, we include KCE, but do not rely strongly on its results to draw conclusions.
\end{itemize}

We acknowledge that this is not a comprehensive list of proposed calibration metrics and we have evaluated metrics such as the Brier score (suggested and highlighted as a proper score in \cite{Gruber2022}) and kernel estimation as proposed in \cite{diciccio2022predictive}, but have omitted them for simplicity as it is difficult to pick a universally "best" CE metric given the current literature. We hope that our work incites further discussion for fairness intervention methods for predictive parity and as a side-consequence, we hope that it encourages the academic community to converge on a definition for calibration error.

\subsection{Detailed Description of Methodologies} \label{App: Methodology list}
In this section, we will provide more in-depth descriptions about the methodologies discussed in Section \ref{section: Case Study}. We will do so in order of how they are presented, omitting certain methodologies as we felt that they are sufficiently described in the main section (e.g., group as a feature, per-group calibrator, tune for worst group calibration, tune for accuracy).

\subsubsection{Group-robust calibration training}
For the neural network implementation, the group-robust calibration training scheme optimizes the following loss function.
\begin{equation}
    \mathcal{L}(x,y,g)=\sum_{i} y_{i}log(f(x_{i}))+(1-y_{i})log(1-f(x_{i}))+\gamma( \underset{g\in\mathcal{G}}{max}E_{(x,y)}\sim\hat{P_{g}}[CE(f)])
\end{equation}
The intuition behind this loss function is that we have a basic loss (binary cross-entropy) that is typically used for binary classification tasks and it is combined with a DRO-style loss that tells the model to focus more on groups where the loss is greater. Formal details (including implementation) of the Group DRO methodology can be found in \citet{sagawa2019distributionally}. The $CE(f)$ term was chosen to be the MMCE kernel as proposed in \citet{kumar2018trainable} and further theoretically analyzed in \citet{BlasiokCalibration}. We repeat it below for completeness:
\begin{equation}
    MMCE^{2}_{m}=\sum_{i,j\in \mathcal(D)}\frac{(c_{i}-r_{i})(c_{j}-r_{j})k(r_{i},r_{j})}{m^{2}} \label{eq: MMCE}
\end{equation}
where $m$ is the number of samples in a group batch and $i,j$ are samples with $c_{i}$ is the correctness score (equals 1 if $\hat{y_{i}}=y_{i}$ and 0 otherwise) and $r_{i}$ is the model outputted score of the sample. The authors show that theoretically, this function is equal to zero only for perfectly calibrated models and intuitively, it applies a dampening force to overconfident predictions that approach 1. It is worth noting that we also attempted other loss functions in our experiments, including focal loss \cite{mukhoti2020calibrating} and Brier score (inspired by the findings in \citet{Gruber2022} which motivates the use of Brier score). However, we found that the that we used worked the most consistently given our experimental setup.

For the XGBoost implementation, we pivoted to using a Brier score loss as the regularization term. The Brier score is also a theoretically backed measure of calibration as it is decomposable into refinement and calibration as a proper score \cite{murphy1986new, Gruber2022}. The reason we use the Brier score instead of MMCE is simply due to the ease of implementation, as XGBoost does not feature automatic gradient/Hessian computations unlike the Torch framework. Brier score has a easily expressible gradient and Hessian and we use these to weigh the samples in each boosting iteration based on the per-group loss relative to the total loss.

\subsubsection{Feature Selection Based On QPD Attribution}
We propose a quantile-based bias metric, Quantile Prediction Drift (QPD), and discuss how to calculate and leverage the feature importance based on QPD to identify which feature(s) could help reduce bias.

A similar metric, Quantile Demographic Drift (QDD), was proposed by \cite{Ghosh_2022}, which uses the model score difference between groups across different score quantiles. The authors proposed that the quantile-based differences of the feature attributions between groups could be used as fairness explanations. Formally,
\begin{equation}
\label{eq:qpd0} QDD_b = E_b[S|G=i] - E_b[S|G=j].
\end{equation}
QDD is closely connected with Demographic Parity (DP). Note that DP can be formally stated as  $$E_b[S|G=i] = E_b[S|G=j] \quad \forall i,j \in \{1,\ldots,G | i\neq j \}.$$
We propose QPD, which is a natural derivation of QDD under the context of PP (equation \ref{eq:pp}). Different from QDD, the metric quantifies the difference in expected \textit{outcome} (instead of expected scores) given any score quantile $b$. For any two groups $i$ and $j$, define
\begin{equation}
\label{eq:qpd1} QPD_b = E[Y | S=s, G=i] - E[Y | S=s, G=j].
\end{equation}
Similar to the derivation in \cite{Ghosh_2022}, QPD can be approximated as
\begin{equation}
\label{eq:qpd2} QPD_b = \frac{1}{N_i}\sum_{n_i=1}^{N_i}Y_{n_i} - \frac{1}{N_j}\sum_{n_j=1}^{N_j}Y_{n_j},
\end{equation}
and the local QPD attribution for feature $f$ (i.e. the feature importance for feature $f$ for $QPD_b$) using feature attribution method A can be calculated as
\begin{equation}
\label{eq:qpda0} QPDA_{b,A,f} = \frac{1}{N_i}\sum_{n_i=1}^{N_i}outcomeAttr_{n_i,A,f} - \frac{1}{N_j}\sum_{n_j=1}^{N_j}outcomeAttr_{n_j,A,f},
\end{equation}
where $outcomeAttr_{n_i,A,f}$ refers to the local attribution of feature $f$ for \textit{outcome} $Y_i$ given instance $n_i$, calculated using feature attribution method A.
For an attribution method A (e.g. SHAP) that satisfies the efficiency axiom \cite{Ghosh_2022}, we can show
\begin{equation}
\label{eq:qpda1} \frac{1}{N_i}\sum_{n_i=1}^{N_i}\sum_{f=1}^{F}outcomeAttr_{n_i,A,f} - \frac{1}{N_j}\sum_{n_j=1}^{N_j}\sum_{f=1}^{F}outcomeAttr_{n_j,A,f} = \frac{1}{N_i}\sum_{n_i=1}^{N_i}Y_{n_i} - \frac{1}{N_j}\sum_{n_j=1}^{N_j}Y_{n_j},
\end{equation}
and therefore,
\begin{equation}
\label{eq:qpda2} QPD_b = \sum_{f=1}^{F} QPDA_{b,A,f}.
\end{equation}
Local feature attribution methods (e.g. SHAP) typically calculates the attribution to model scores $S$. In order to calculate the attribution to another random variable $T$, we generally need access to a data generation mechanism $g(X_w)=T_w$, which tells us the ground-truth value of $T$ given the feature values of any hypothetical instance $w$.
One way to approximate this mechanism is with a superior model. Under the use case of feature selection, we assume that we are able to train a less biased and more accurate model $M_s$ with the additional features, compared to the original model $M_o$, so that $\hat{g} (X_w) = M_s (X_w) = S_{M_s}$ can be a proxy to the outcome $Y_w$. Although the approximation is not perfect, we can use it to identify features which attributes to the bias difference between $M_s$ and $M_o$, and this is more effective when $M_s$ is less biased.

When we have a large number of available features to be added to the model, it is often more likely that we can build a better model $M_s$ that is closer to the ground truth outcomes; when the number of available features is large, since the alternative methods (e.g. brute-force retraining with all possible feature combinations, or add all features to the model) would be extremely expensive, the feature selection mechanism that we propose would also be more useful.

In our experiments, we identified the 3 (out of 7) worse calibrated demographic groups in terms of ECCE. We defined two groups for QPD attribution: one group contains all instances within the worse calibrated demographic groups, and the other contains all other instances. We calculated QPD attribution using SHAP and the above formulation to identify the importance features related to bias. To mimic realistic scenarios where building and maintaining feature pipelines are expensive, we identify \textit{one} top feature with the highest aggregated QPD attribution to be added to the model.

Based on experiment results, adding the one top feature identified with QPD attribution often has similar effect as adding all features to the model. We conclude that QPD attribution is an efficient way to identify which feature(s) could help achieve better PP.

\subsubsection{Multicalibration}
This methodology was first proposed in 2018 \cite{pmlr-v80-hebert-johnson18a} as a post-processing approach to achieve algorithmic fairness. It has two key assumptions: 1) Bias comes from the model fitting procedure that optimizes for global accuracy, leaving the minority groups under-fitted. However, the training data and labels are unbiased. 2) When the protected attribute is not accessible, there are other correlated features that can jointly identify the subpopulations. The way it works is to iteratively calibrate the original predictor on validation data, partition by partition, until no identifiable subgroup has miscalibration larger than a predefined threshold or the maximum number of iterations has reached. Then the calibrated predictor can be applied to any new dataset with the same features. We evaluated the effectiveness of multicalibration and its variants on reducing worst-group calibration error (ECCE), without using the protected attribute at any stage.

There are many possible options to configure the exact method of running multicalibration. Instead of hand-picking, we set them as hyperparameters to tune, which allows the best combination to emerge for each use case. Specifically, these hyperparameters fall into the following categories.

(1) Partition strategy: The total number of partitions and how to create them can be tuned. Probability ranges can split evenly between 0 and 1, or split by quantile to have equal number of instances within each partition.

(2) Sampling strategy: At each iteration, without sampling it uses the same data to fit on the residuals. To reduce over-fitting, we could either split the data equally such that every iteration uses a disjoint subset, or perform bootstrapping to introduce some variation.

(3) Training algorithm: Calibration relies on the predicted residuals. At each iteration, we train a ridge regression or decision tree regression model using all the features to predict the remaining residuals. To reduce over-fitting, the regularization strength and the maximum depth of tree can be tuned, respectively.

(4) Probability update strategy: Miscalibration is estimated by the correlation between predicted and observed residuals. At each iteration, only the partition with the largest miscalibration will get an update on the probability scores. Multicalibration \cite{pmlr-v80-hebert-johnson18a} applies an additive weight to the original probability scores, and re-partition the data based on the new probability scores. A slight variant, multiaccuracy \cite{DBLP:journals/corr/abs-1805-12317}, applies a multiplicative weight to the original probability scores, and does not re-partition the data. In addition, the step size for probability update can also be tuned.

(5) Stopping condition: It triggers a stop when the largest miscalibration across all the partitions is smaller than a predefined threshold, or the maximum number of iterations has been reached. Both the threshold and the maximum number of iterations can be tuned.

For each use case, we took 70\% of the validation data to run the algorithm. During the execution, which partition got updated and its corresponding residual model are both stored for all the iterations. This was done for 20 trials with different combinations of hyperparameters. After that, we replayed the algorithm on the rest 30\% of validation data to select the best configuration. The exact version of algorithm was then evaluated on the test dataset.

\subsubsection{Detailed Description of Calibration Methods} \label{App: Calibration Methods}

The magnitude of the model score can be seen an estimate of confidence in the prediction for classification tasks. Post-hoc calibration methods apply a confidence mapping $g$ on top of a mis-calibrated scoring classifier $\hat{p} = h(x)$ to deliver a calibrated confidence score $\hat{q} = g(h(x))$, in order to reduce or eliminate calibration errors \cite{Kueppers_2020_CVPR_Workshops}.

Below we describe each of the calibration methods that we tested in the experiments:

\begin{itemize}
  \item \textbf{Histogram Binning}: Proposed by \cite{Zadrozny2001ObtainingCP}, histogram binning sorts each prediction into a bin and assigned its calibrated confidence estimate. Specifically, the probability that an example belongs to a class is estimated with the fraction of training examples in the bin that actually belong to the calss. To extend this method to multiclass classification, we apply the method in a 1-vs-all fashion \cite{Kueppers_2020_CVPR_Workshops}.
  \item \textbf{Isotonic Regression}: Isotonic Regression is proposed by \cite{isotonicReg}. This method is similar to Histogram Binning, except that it uses dynamic bin sizes as well as boundaries. It fits a piece-wise constant function to the ground truth labels sorted by given confidence estimates.
  \item \textbf{Bayesian Binning into Quantiles (BBQ)}: Proposed by \cite{naeini2015bbq}, this method utilizes several Histogram Binning instances with different number of bins. It then obtains the calibrated confidence estimate with a weighted sum of these instances. The scoring function is modified per \cite{Kueppers_2020_CVPR_Workshops}.
  \item \textbf{Ensemble of Near Isotonic Regression (ENIR) models}: Proposed by \cite{naeini2016ENIR}, this method allows for a violation of the monotony restrictions, which is different from Isotonic Regression. ENIR uses the modified Pool-Adjacent-Violators Algorithm (mPAVA) and builds multiple Near Isotonic Regression models. Similar to BBQ, it then uses a weighted score function based on these model instances.
  \item \textbf{Logistic Calibration}: Logistic Calibration is also called Platt Scaling. It is proposed by \cite{Platt2007ProbabilisticOF}. This method uses multiple independent normal distributions to obtain a calibration mapping by means of the confidence and additional features. Specifically, $$g(s) = \frac{1}{1+exp(-z(s))}$$ where $z(s)=lr(s)=s^Tw+c$ is a logistic regression. This calibration scheme assumes independence between all variables.
  \item \textbf{Beta Calibration}: Proposed by \cite{Kull2017BetaCA}, this method is similar to Logistic Calibration except for a different $lr(s)$ formulation. It uses multiple independent Beta distributions to obtain a calibration mapping by means of the confidence as well as additional features. This calibration scheme assumes independence between all variables.
  \item \textbf{Temperature Scaling}: This method is proposed by \cite{guoCalibration}. It divides the logits (inputs to the softmax function) by a learned temperature scaling scalar parameter $T$, so that $$\hat{q} = \sigma(\frac{s}{T})$$ where $s=(\hat{p}, \hat{r})$ is a combined input and $\hat{r} \in {[0,1]}^J$ is an additional box regression output.
  \item \textbf{PlattBinner}: This method is proposed by \cite{Kumar2019} to address the issue that the calibration error of methods like Platt scaling and temperature scaling are typically underestimated and cannot be measured, and the calibration error of methods like Histogram Binning requires many examples to measure. The authors propose that PlattBinner is an efficient recalibration method where the calibration error can be measured with fewer examples.
\end{itemize}

\subsection{Results for MSCE}  \label{App: Results for MSCE}
Our study when evaluated on MSCE yields very similar conclusions to our evaluation of NN. We show the Pareto optimality Tables (Table \ref{tbl: XGBsummaryMSCE}, Table \ref{tbl: NNsummaryMSCE}) as well as the aggregate results over the ACS datasets and the aggregate results in Figure  \ref{fig:MSCEFrontiers}. Note that because we tuned some of the methods for ECCE and are evaluating them for MSCE and KCE in the respective sections, the comparisons between the calibration error metrics are not entirely sensible but we show them anyways for comprehensiveness.

\begin{center}
\begin{table}[h]
\begin{tabulary}{\linewidth}{||LCCL||}
 \hline
 Data Availability & \# Methods Tested & \# Times Optimal Across Datasets & Best Pareto Optimal Method \\ [0.5ex]
 \hline\hline
 Train+Val+Inf & 2 & 7 & Use group (5/7) \\
 \hline
 Train+Val & 1 & 3 & Group-robust calibration training (3/3) \\
 \hline
 Val & 3 & 7 & Group-robust calibrator (4/7) \\
 \hline
 None & 10 & 10 & Tune for accuracy (3/10) \\ [1ex]
 \hline
\end{tabulary}
\caption{Summarizing benchmarks across 10 datasets for XGB (Using MSCE)}
 \label{tbl: XGBsummaryMSCE}
\end{table}
\end{center}

\begin{center}
\begin{table}[h]
\begin{tabulary}{\linewidth}{||LCCL||}
 \hline
 Data Availability & \# Methods Tested & \# Times Optimal Across Datasets & Best Pareto Optimal Method \\ [0.5ex]
 \hline\hline
 Train+Val+Inf & 2 & 8 & Use group (4/8) \\
 \hline
 Train+Val & 2 & 7 & Group-robust calibration training (4/7) \\
 \hline
 Val & 3 & 1 & Group-robust calibrator (1/1) \\
 \hline
 None & 12 & 10 & Multicalibration (4/10) \\ [1ex]
 \hline
\end{tabulary}

\caption{Summarizing benchmarks across 10 datasets for NN (Using MSCE)}
 \label{tbl: NNsummaryMSCE}
\end{table}
\end{center}

\begin{center}
    \begin{figure}[h]
    \caption{Pareto frontiers when aggregating data across ACS data (top) and All data (bottom)}
    \label{fig:MSCEFrontiers}
    \includegraphics[width=\textwidth]{Images/ParetoAggregateACSMSCE.png}
    \includegraphics[width=\textwidth]{Images/ParetoAllMSCE.png}
    \end{figure}
\end{center}

\subsection{Pareto Frontier Tables for KCE} \label{App: Results for KCE}

In this section, we show the same thing as in Appendix \ref{App: Results for MSCE}. Again, we note that because we tuned some of the methods for ECCE and are evaluating them for KCE, the comparisons between the calibration error metrics are not entirely sensible but we show them anyways for comprehensiveness. In \ref{App: CE Metrics} we also remarked that we consider KCE to be the least reliable calibration error metric, as although it has intriguing theoretical properties \cite{BlasiokCalibration} practically evaluating this metric in a reasonable amount of time calls for random sampling, which we find to be an undesirable property in general and particularly undesirable when making sensitive measurements in real applications such as fairness. We show the Pareto frontier tables in Table \ref{tbl: XGBsummaryKCE} and Table \ref{tbl: NNsummaryKCE} as well as the aggregate results over the ACS datasets and the aggregate results in Figure \ref{fig:KCEFrontiers}.

\begin{center}
\begin{table}[h]
\begin{tabulary}{\linewidth}{||LCCL||}
 \hline
 Data Availability & \# Methods Tested & \# Times Optimal Across Datasets & Best Pareto Optimal Method \\ [0.5ex]
 \hline\hline
 Train+Val+Inf & 2 & 7 & Use group (5/7) \\
 \hline
 Train+Val & 1 & 3 & Group-robust calibration training (3/3) \\
 \hline
 Val & 3 & 7 & Group-robust calibrator (4/7) \\
 \hline
 None & 10 & 10 & Tune for accuracy (3/10) \\ [1ex]
 \hline
\end{tabulary}

\caption{Summarizing benchmarks across 10 datasets for XGB (Using KCE)}
\label{tbl: XGBsummaryKCE}
\end{table}
\end{center}

\begin{center}
\begin{table}[h]
\begin{tabulary}{\linewidth}{||LCCL||}
 \hline
 Data Availability & \# Methods Tested & \# Times Optimal Across Datasets & Best Pareto Optimal Method \\ [0.5ex]
 \hline\hline
 Train+Val+Inf & 2 & 8 & Use group (4/8) \\
 \hline
 Train+Val & 2 & 7 & Group-robust calibration training (4/7) \\
 \hline
 Val & 3 & 1 & Group-robust calibrator (1/1) \\
 \hline
 None & 12 & 10 & Multicalibration (4/10) \\ [1ex]
 \hline
\end{tabulary}

\caption{Summarizing benchmarks across 10 datasets for NN (Using KCE)}
\label{tbl: NNsummaryKCE}
\end{table}
\end{center}

\begin{center}
    \begin{figure}[h]
    \caption{Pareto frontiers when aggregating data across ACS data (top) and All data (bottom)}
    \label{fig:KCEFrontiers}
    \includegraphics[width=\textwidth]{Images/ParetoACSKCE.png}
    \includegraphics[width=\textwidth]{Images/ParetoAllKCE.png}
    \end{figure}
\end{center}

\section{A Primer on Predictive Parity Fairness}
For our case study, we will focus on a particular group fairness definition known as predictive parity (hereafter abbreviated as PP), also known as sufficiency. In this section, we first provide background on PP and motivate its importance. Then, we describe how and why we measure PP through the worst-group calibration error of a model. Lastly, we provide an overview of methods for achieving PP that recaps well-known methods and also introduces new methods inspired by robust optimization principles.

\subsection{Honing in on Predictive Parity}
PP as a definition was popularized in 2017 in the context of the COMPAS Recidivism study \cite{dieterich2016compas}\cite{Chouldechova17}. In words, this condition states that a model predicted score should have the same interpretation of outcome likelihood regardless of who the prediction is being made for. Formally, it states that for a classifier that produces a risk score $s\in [0,1]$ to predict an observation's outcome propensity $Y\in\{0,1\}$ where the observation's group membership $g\in \mathcal{G}=\{1,\dots,G\}$, we must have the following:
\begin{equation}
\label{eq:pp} E[Y|S=s, G=i] = E[Y|S=s, G=j] \quad \forall i,j \in \{\mathcal{G}| i\neq j \}
\end{equation}

PP is closely related to notions of equal calibration \cite{Chouldechova17,barocas-hardt-narayanan}, which is a stronger notion that states that the score produced by a model is representative of the empirical likelihood of the event it is predicting, regardless of who the prediction is for. Formally, this is the condition that:
\begin{equation}
\label{eq:equalCal} E[Y|S=s, G=g]=s \quad \forall g\in \mathcal{G}
\end{equation}
We will return to this connection in the next subsection but for now reference equal calibration to motivate the importance of achieving PP. There are several reasons why PP is a desirable fairness property in a variety of domains. First, it is a notion of fairness that is easily justifiable as it does not explicitly harm or benefit one group, but rather tries to ensure that scores are reflective of empirical risk. Second, as ML models are often used in junction with human decision-makers, equal calibration ensures that model outputs are interpretable and comparable. Lastly, calibration is a desirable property when scores are used in safety-critical applications or when used in downstream purposes.

\subsection{Predictive Parity and Calibration} \label{subsection: PP and Cal}
As elaborated in \cite{barocas-hardt-narayanan}, there is a clear linkage between PP and calibration. Namely, a model that satisfies PP can be made calibrated through a simple output mapping transformation, and that a calibrated model satisfies PP. This linkage thus provides a straightforward method to achieving PP by simply calibrating the model. This approach is favored as it also brings the aforementioned benefits of calibration. Hence, if we accept that calibration is the most intuitive strategy to achieve PP, then evaluating for PP fairness boils down to ensuring that each group is equally well calibrated. In other words, we aim to ensure that the fairness intervention for predictive parity through calibration worked equally well for all groups.

Thus, we define PP in the multiple group setting for our study by looking at the measuring the calibration error (CE) of the worst-calibrated group on the testing data $\mathcal{D}$.
\begin{equation}
    Worst \: group \: CE = \underset{g\in\mathcal{G}}{max}\ CE_{(s,y,g)\sim \mathcal{D}}(f) \label{eq: wgce}
\end{equation}
This is as opposed to the average or overall CE: $CE_{(s,y,g)\sim \mathcal{D}}(f)$, which may overlook the (in)effectiveness of the fairness intervention to any particular group. As for which CE we will use, we opt for the Expected Cumulative Calibration Error (ECCE) \cite{arrieta2022metrics}. This metric was selected over the commonly-used expected calibration error (ECE) \cite{guoCalibration} after we surveyed recent developments in CE metrics and found several works pointing out that the had statistical and empirical flaws. We recap these arguments in detail in Appendix \ref{App: CE Metrics} and summarize our case study for two other CE metrics we considered as well (the mean sweep calibration error (MSCE) of \cite{roelofs2022mitigating} and the kernel calibration error (KCE) of \cite{BlasiokCalibration}) in Appendix \ref{App: Results for MSCE}, \ref{App: Results for KCE}. With our goal established as investigating methods for minimizing worst-group ECCE for predictive parity fairness, we now turn to our case study.

\subsection{Methods for Achieving Predictive Parity}

Existing works for obtaining PP include \cite{pleiss2017fairness} which attempts to maintain calibration of a model alongside equalized odds, \cite{pmlr-v80-hebert-johnson18a} which presents a individual-fairness notion of calibration called multicalibration, \cite{Hsu2022} which uses group membership in post-processing for PP but does not calibrate, \cite{LeeSufficiency} which regularizes for PP using an information-theoretic approach, and \cite{zeng2022sufficiency} which applies to binary $0/1$ classifiers.

In our review of algorithms and recent surveys \cite{HortMitigationSurvey}, methods for achieving PP does not appear to receive as much focus as other group fairness definitions such as demographic parity and equalized odds \cite{Hardtetal_NIPS2016} as also observed in \cite{LeeSufficiency}, \cite{zeng2022sufficiency}. The relative lack of attention may be due to "obviousness" of the solution, being to simply calibrate the model separately on each group or to use the group attribute. Moreover, there is theoretical evidence showing that PP essentially comes for free in unconstrained learning situations when the model features have enough predictive power to infer group membership \cite{LiuImplicitFairnessCriterion}. The relative ease of achieving PP when group membership is available as a feature has perhaps led to less attention on bias mitigation strategies. However, we believe that despite these arguments, further investigation into achieving PP is warranted. Firstly, as a central theme to this work, group data is not always available as a model feature due to optical or legal reasons. Moreover, even if it is available, there are reasons to avoid its usage. Secondly, the conditions mentioned in \cite{LiuImplicitFairnessCriterion} may not always be met as many commonly used models are \textit{not} trained with unconstrained loss minimization (e.g. regularized training) nor are features always informative of group membership. Hence, our work also aims to achieve PP without full group data access.

\begin{center}
    \begin{figure}[h]
    \caption{Methods benchmarked in our case study analyzed by our proposed four pillars}
    \label{fig:POI}
    \includegraphics[width=\textwidth]{Images/POIFig.png}
    \end{figure}
\end{center}

Towards this goal, our benchmarks cover both fundamental/well-known solutions for achieving predictive parity (e.g., model calibration, tuning for calibration) as well as novel in-processing methods that we have devised (group-robust calibration). We depict these methods from the lens of our four-pillar framework in \ref{fig:POI}. Our methods focus on achieving PP in two models that are commonly used in industry for their consistently strong performance on tabular data - Extreme Gradient Boosting (XGB) \cite{Chen:2016:XST:2939672.2939785} and feed-forward neural networks (NN). Due to the large suite of methods we benchmark, we will provide a brief description of each methodology below, in decreasing order of data requirements and provide thorough descriptions with implementation details in the appendix. The ordering is meant to reflect the increasing "acceptableness" of the mitigation approach, as avoiding the use of demographic/protected attribute data is generally seen as more palatable according to the multiple desiderata discussed above. Another theme of these methods is the contrast of addressing average-case calibration error by avoiding use of group data versus addressing worst-group calibration error at the expense of requiring some extent of group data.

\begin{itemize}
    \item \textbf{Train + Validation + Inference}: In this regime, the protected attribute data is available for use in all steps of the model pipeline.
    \begin{itemize}
        \item \textbf{Group as a feature} (In): We encode the group attribute as indicator variables (one for each group) and use them in the model at all steps.
        \item \textbf{Per-group calibrator} (Post): After a model is tuned for accuracy, we use it to predict probabilities on the validation set and use the scores and labels to fit a post-hoc calibrator (isotonic regression \cite{MizilPlattIsotonic}) separately for each group. At inference, we use the group attribute of each observation to select the respective model for calibrating the scores. This explicitly aims to optimize for \ref{eq: wgce}.
    \end{itemize}
    \item \textbf{Train + Validation}: In this regime, the protected attribute is available in the training and tuning phase, but not accessible for inference.
    \begin{itemize}
        \item \textbf{Group-robust calibration training} (In):  We modify the objective of the gradient boosting and neural network models to simultaneously optimize the base loss (cross entropy) and the worst-group calibration loss $CE(f)$:
        \begin{equation}
            \mathcal{L}(x,y,g)=\sum_{i} y_{i}log(f(x_{i}))+(1-y_{i})log(1-f(x_{i}))+\gamma( \underset{g\in\mathcal{G}}{max}E_{(x,y)}\sim\hat{P_{g}}[CE(f)])
            \label{eq: droCalLoss}
        \end{equation}
        For neural networks, the DRO loss \cite{sagawa2019distributionally} is applied on top of the Maximum Mean Calibration Error (MMCE) loss from \cite{kumar2018trainable}. For XGB, the DRO loss\footnote{We use DRO here loosely, as the exact mirror descent algorithm proposed in \citet{sagawa2019distributionally} can only be heuristically replicated in the XGB framework.} is applied on top of Brier score which can be decomposed into calibration and reliability \cite{murphy1986new, Gruber2022}.
    \end{itemize}
    \item \textbf{Validation}: In this regime, the protected attribute is available for tuning purposes or training models with very few hyperparameters (such as calibrators that train on one variable).
    \begin{itemize}
        \item \textbf{Tune for accuracy} (In): For each set of hyperparameters, we evaluate the resulting CE on the validation data and pick the set of parameters that yields the smallest worst-group CE.
        \item \textbf{Group-robust Calibrator} (Post): We fit a boosting model except instead of using the original data features, the only feature is the predicted score from the base (uncalibrated) model. The model is trained with the mixture group-robust loss function.
    \end{itemize}
    \item \textbf{None}: In this regime, we do not have the protected attribute at any stage.
        \begin{itemize}
            \item \textbf{Calibration loss} (In): The default choice for binary prediction loss functions is cross entropy/logistic loss. However, there are other loss functions that fare better for calibration purposes that have been explored in NNs. These include temperature scaling \cite{guoCalibration}, focal loss \cite{mukhoti2020calibrating}, etc. (see \cite{minderer2021revisiting} for a recap of related calibration techniques for neural networks). This has not been as well-studied in boosting frameworks. However, \cite{Gruber2022} shows that the Brier score is a reasonable choice of loss function for achieving calibration (defined across a variety of CE metrics).
            \item \textbf{Multicalibration} (Post): Multicalibration \cite{pmlr-v80-hebert-johnson18a} and its variant Multiaccuracy \cite{DBLP:journals/corr/abs-1805-12317} have shown promises on improving multi-group fairness by iteratively calibrating subpopulations that are defined by the intersections of features. Presumably, it works better when there exist features which correlate with the protected attribute. We implemented various options of this approach as hyperparameters to tune (see Appendix), and selected the best option to test.
            \item \textbf{General calibrators} (Post):  Post-hoc calibrators apply a confidence mapping $g$ on top of a mis-calibrated scoring classifier $\hat{p} = h(x)$ to deliver a calibrated confidence score $\hat{q} = g(h(x))$, to improve (average-case) calibration on the \textit{overall} population \cite{Kueppers_2020_CVPR_Workshops}. This is in contrast to the per-group calibrator mentioned above which customizes the calibration for each group. These methods include histogram calibration, isotonic regression \cite{MizilPlattIsotonic}, Platt scaling \cite{Platt99probabilisticoutputs}, Bayesian Binning into Quantiles (BBQ) \cite{naeini2015bbq}, Ensemble of Near-Isotonic Regressions (ENIR) \cite{naeini2016ENIR}, Beta calibration \cite{kull2019betaScaling}, and the Platt scaling and histogram binning hybrid as developed in \cite{Kumar2019}. Please see \ref{App: Calibration Methods} for detailed descriptions. We test all of the above calibrators based on the implementations made available by in the Netcal Python package \cite{Kueppers_2020_CVPR_Workshops}.
        \end{itemize}

\end{itemize}

\paragraph{Feature Selection Methods for Predictive Parity}
In the intervention methods discussed above, we have excluded examining the highly practical choice of making the model better by finding more/better features, which has been explored in literature \cite{CaiFairAllocation, Ghosh_2022}. One may even argue that adding features linked to group data presents a theoretically backed strategy towards PP fairness based on the findings of \citet{LiuImplicitFairnessCriterion}. However, as mentioned in Section \ref{Section: Framework} in the context of engineering efforts, we find that most large companies pursing AI fairness operate via a centralized responsible AI team. This means that finding features for each model that both improve performance and fairness can be greatly labor intensive and difficult to scale across many models.

Yet, in reality where scaling up responsible AI efforts is not a priority (e.g., only a few consumer-facing models need to be made fairer), this is a reasonable option to pursue and we present a novel feature-selection based method towards PP fairness. In many scenarios, we often have other non-demographic data used in other production systems in the organization but are not included in a particular model due to being deemed as irrelevant or was phased out in the feature selection step. Some of such features, though it may not aid in accuracy, may improve fairness. Given that including all features is often not realistic (due high cost of collection or pipeline maintenance) we aim to select the right feature(s) to use in our model. We propose a quantile-based bias metric, Quantile Prediction Drift (QPD), which is derived from the Quantile Demographic Drift (QDD) bias metric proposed by \citet{Ghosh_2022}. We show that QPD feature attributions is an efficient way to help find features that could help achieve PP if included in the model. Due to the greater dependence on context for this methodology, we leave the details and experimental results in Appendix \ref{App: Methodology list}.
\section{Case Study} \label{section: Case Study}

Having set up the fairness metric of interest (predictive parity), defined its measurement (worst-group calibration error), and established several methodologies to benchmark, we now move on to our case study. To first summarize, our experiments will show the following results:
\begin{enumerate}
    \item \textbf{Usage of group data appears to be necessary for achieving PP fairness}. We find that methods that do not use any group data (e.g., average-case calibrators and multicalibration) do not provide much benefit over the baseline of tuning for accuracy.
    \item \textbf{Group-robust optimization in training offers a promising path toward PP fairness}. We find that our proposed DRO-based methods are consistently Pareto optimal in the sense that no method in our benchmarking can achieve better fairness without requiring more group data. These findings point to further research opportunities that to our knowledge have not been explored for PP fairness.
    \item \textbf{Model inductive bias is a key part of fairness}. We find that XGBoost consistently outperforms NN on all fronts (accuracy, fairness, ease of implementation) on tabular data. This emphasizes that the base model choice may have significant impacts on fairness properties and thus should not be overlooked.
\end{enumerate}

\subsection{Overview}

Our proposition for operationalizing fairness interventions is to quantitatively and qualitatively consider the trade-offs between four pillars of (1) data availability (2) place of intervention and engineering effort (3) intervention effectiveness (4) model performance. Of these, we find (1), (3), and (4) to be the most quantifiable and demonstrate our framework by quantifying the trade-offs between these factors, with model performance acting as a guardrail. In applying our framework in real situations, engineering effort would have to be considered but given that its quantification is organization and team-dependent, we leave it out of the evaluation for now.

We focus our testing on methods to achieve predictive parity through calibration in binary classification settings with tabular data. The overarching process for these experiments is as follows and for each data set considered, we run these steps over three trials to produce means and standard deviations for all reported metrics:

\begin{enumerate}
    \item Split the data randomly into training-validation-testing data sets using a 60/20/20 split.
    \item Tune a base model (XGB or neural network) by tuning hyperparameters over a fixed grid to optimize for a particular metric. This is done by fitting on the training data and evaluating hyperparameter quality on the validation data.
    \item Use the best set of hyperparameters found in the previous step to fit the base model on the training data.
    \item Apply the predictive parity mitigation technique(s).
    \item Predict and evaluate the effectiveness of the mitigation method(s) on the testing data.
\end{enumerate}

\subsection{Experiment results}

Through our experiments, we can produce a Pareto frontier that depicts the trade-offs between data requirements and intervention effectiveness. In this section, we will first demo and analyze results from the ACS datasets to highlight some overarching themes and then distill the results across all 10 datasets after. We will report equivalent statistics when defining bias on the basis of worst-case MSCE and KCE in the appendix, but for now note that the results are generally aligned (especially between ECCE and MSCE).

\begin{center}
    \begin{figure}[h]
    \caption{The Pareto frontier of fairness interventions for two model classes (XGBoost and Neural Networks) aggregating results over datasets. The degree of predictive parity fairness is plotted on the Y-axis and data requirement on the X-axis. Each point represents a unique fairness intervention method for predictive parity (points close to the bottom left-hand corner are better). We have labeled the Pareto optimal points as well as the baseline points "Tune for Acc." Model performance is considered by observing the accuracy range.}
    \label{fig:Aggregate ACS Frontier}
    \includegraphics[width=\textwidth]{Images/ParetoACSECCE.png}
    \end{figure}
\end{center}

We first illustrate this frontier in the Income Poverty dataset with the XGB and NN models (Figure \ref{fig:Aggregate ACS Frontier}). Comparing the two model classes first, we observe that the Pareto frontier for the XGB model lies entirely below the corresponding line and furthermore the minimum overall accuracy (shown in the subtitle of Figure\ref{fig:Aggregate ACS Frontier}) across all methods for XGB is higher than the maximum overall accuracy than NN models. This demonstrates the importance of the inductive biases and reinforces the findings of \cite{GardnerXGB}, which underscores the extremely hard-to-beat performance of XGBoost on tabular data regardless of what we are measuring.

Next, we look at similarities across both models. We first expectantly observe that more data availability tends to yield lower bias, which highlights the importance of our data requirement pillar. We also observe that despite the wide array of interventions attempted, the overall accuracy does not vary by much. This is a known aspect of predictive parity fairness, whereby unlike other notions such as demographic parity/equalized odds, inducing predictive parity has relatively muted effects or can even improve accuracy \cite{DurfeeHetCal}. More interestingly, we observe that some intuitive solutions for addressing predictive parity do not work as well as one may initially expect. For instance, while it may be intuitive that a post-hoc calibrator such as isotonic regression or Platt scaling may achieve much better worst-group calibration, it turns out these methods are not much better than optimizing for accuracy. This may be due to the fact that the worst-case performance is often observed in the smallest group and thus while calibrators may shrink overall calibration error, it does little to no favors for the worst group and thus fails to achieve PP fairness.

Next, we will rely on notions of Pareto optimality to aggregate the results over our 10 datasets (Tables \ref{tbl: XGBsummary}, \ref{tbl: NNsummary}) and distill them into a set of conclusions. For each dataset, we will first identify the best method in each data availability category (None, Val, Train+Val, or Train+Val+Inference) and determine if it is Pareto optimal (with respect to the multi-objective optimization problem of minimizing worst-group calibration error and minimizing use of group attributes). Then, we will look across the 10 datasets and count the number of times a category produced a Pareto optimal method (third column of Tables \ref{tbl: XGBsummary}, \ref{tbl: NNsummary}). Finally, conditional on a data availability category having a Pareto optimal method, we report the most frequently optimal method across all such instances (fourth column of Tables \ref{tbl: XGBsummary}, \ref{tbl: NNsummary})\footnote{To give an example, row 3 of Table \ref{tbl: XGBsummary} says that in 7 out of 10 datasets, getting lower worst-group calibration error than the best method of Val data only requires using more group data. Of those 7 instances, the group-robust calibrator method is the best perfoming in 4/7 of them.}.

\begin{center}
\begin{table}[h]
\begin{tabulary}{\linewidth}{||LCCL||}
 \hline
 Data Availability & \# Methods Tested & \# Times Optimal Across Datasets & Best Pareto Optimal Method \\ [0.5ex]
 \hline\hline
 Train+Val+Inf & 2 & 7 & Use group (5/7) \\
 \hline
 Train+Val & 1 & 3 & Group-robust calibration training (3/3) \\
 \hline
 Val & 3 & 7 & Group-robust calibrator (4/7) \\
 \hline
 None & 10 & 10 & Tune for accuracy (3/10) \\ [1ex]
 \hline
\end{tabulary}
\caption{Summarizing benchmarks across 10 datasets for XGB}
\label{tbl: XGBsummary}
\end{table}
\end{center}

\begin{center}
\begin{table}[h]
\begin{tabulary}{\linewidth}{||LCCL||}
 \hline
 Data Availability & \# Methods Tested & \# Times Optimal Across Datasets & Best Pareto Optimal Method \\ [0.5ex]
 \hline\hline
 Train+Val+Inf & 2 & 8 & Use group (4/8) \\
 \hline
 Train+Val & 2 & 7 & Group-robust calibration training (4/7) \\
 \hline
 Val & 3 & 1 & Group-robust calibrator (1/1) \\
 \hline
 None & 12 & 10 & Multicalibration (4/10) \\ [1ex]
 \hline
\end{tabulary}
\caption{Summarizing benchmarks across 10 datasets for NN}
\label{tbl: NNsummary}
\end{table}
\end{center}

We now synthesize these results into conclusions and recommendations for practitioners.

\paragraph{\textbf{Usage of group data appears to be necessary for achieving PP fairness}}
Focusing first on the two extremes - full availability and no availability, the clearest result is that the most consistent and effective way to achieve PP fairness is to use the group attribute as a feature (though whether this \textit{should} be done is a separate discussion which we will return to in the conclusion). This result is not surprising, but it does hint that theoretical results showing that PP fairness comes for free under sufficiently rich models \cite{LiuImplicitFairnessCriterion} should not be overly relied on in practice, as there are several commonly used models (XGB, neural networks) that still benefit largely from the use of group data. On the other end of the spectrum, when no group data is available, we find that average-case calibrators and even multicalibration cannot consistently beat the baseline of simply tuning for accuracy. This is likely due to the fact that average-case calibrators mostly improve calibration for the largest group, which is typically not the group with the worst-case calibration error. We summarize these observations in the following remark.

\begin{remark}
    The lack of empirical effectiveness across several mitigation strategies that avoid using group data indicates that practitioners seeking to achieve PP may need at least some amount of group data to reliably fix gaps in predictive parity.
\end{remark}

\paragraph{\textbf{Group-robust calibration offers a promising path toward PP fairness}}
Turning to the results in the val and train+val data availability regimes, we find that our proposed methods of using a group-robust calibrator or group-robust calibration training show strong promise. In particular, the group-robust calibration training (\ref{eq: droCalLoss}) works consistently well in achieving PP fairness for the neural network model\footnote{The method (or the method with temperature scaling) was Pareto optimal in 6/6 ACS datasets, but did not work as well in the small datasets: German, Bank, COMPAS, perhaps due to the lack of signal in the smaller data.}. This success is likely due to the effectiveness of group DRO \cite{sagawa2019distributionally} and MMCE \cite{kumar2018trainable, BlasiokCalibration} being a theoretically-backed measurement of calibration error. The equivalent heuristic technique for XGBoost was not as effective likely due to having a less theoretically principled implementation. However, Pareto dominance on 3 datasets indicates that customizing losses in XGBoost to focus on group-robust calibration loss deserves further investigation. Overall, we emphasize that these are techniques that effectively improve PP fairness \textit{without using group data at inference}.

\paragraph{\textbf{Model inductive bias is a key part of fairness}}
As shown in Figure \ref{fig:Aggregate ACS Frontier}, the frontier of XGBoost sits below that of the NN model and furthermore the best accuracy of the NN models is lower than the worst accuracy across the XGBoost models. To top it off, we found that XGBoost was much easier to work with (in terms of training and tuning) while setting up these large-scale experiments. This is in line with a recent study of subgroup fairness and robustness properties of XGBoost models \cite{GardnerXGB} in which the authors found that on tabular data, XGB tuned for accuracy is a very hard-to-beat baseline (though the study did not evaluate for PP fairness).

\section{Conclusion}

Recent literature has made it clear that AI fairness is a critical problem that absolutely requires collaboration between AI fairness practitioners as well as researchers on both the algorithmic and social/legal theory. Our works overall aims to help bridge this collaboration more closely by presenting a holistic framework that captures social/legal factors (use of group data) and industrial factors (engineering cost/scalability) on top of canonical algorithmic fairness versus performance trade-offs. Our framework emphasizes that the trade-offs between these four pillars can be quantitatively evaluated by surveying the landscape of fairness algorithms and further considering the data requirements for each algorithm rather than just the standard pre-/in-/post-processing categorization. As we demonstrate with a case study on predictive parity fairness, we hope that illustrating these trade-offs can further fill the gap in communications between system-level directives and researchers on the algorithms side. Our case study also yields methodological findings on achieving predictive parity fairness by shedding light on the need for use of group data, the effectiveness of group-robust calibration training, and the role of model inductive bias in fairness.

Beyond our contributions, we hope that this work adds to the growing movement of practical considerations for fairness. Throughout the legal and social literature that we have reviewed (\cite{AndrusDemographic, AndrusSpitzer, BogenAwareness, bent2019algorithmic, xiang2020reconciling}), the only consistent theme is that there does not seem to be a hard and fast rule for the collection and usage of group data. We therefore urge academics and practitioners to continually monitor the developments of AI fairness in society, law, and industry to research and implement solutions accordingly.
\section{Holistic Fairness Intervention Selection Framework} \label{Section: Framework}
We now propose our framework for operationalizing fairness interventions in practice. The key objective of the framework is to tie the system-level objectives and constraints (e.g., degree of access to group data, engineering effort and budget) to the selection of the right fairness intervention algorithm. This is achieved by emphasizing two major aspects of fairness intervention---the \emph{how} and the \emph{where}. They dictate the class of the candidate algorithms and encapsulate both quantitative (data size/availability and intervention model complexity) and qualitative (engineering effort and how transparent  the intervention is to stakeholders) considerations in ML development. We will present our framework in four pillars, with the first two focusing on these operationally-focused aspects and the latter focusing on the canonical considerations of fairness versus average predictive performance trade-offs. In describing our proposed evaluation guidelines, we focus our guidance for usage in large-scale online platforms; we expect our development will often have a bigger bearing on non-technology companies with less sophisticated engineering infrastructures. Note that while the choice of which fairness definition to use is a major aspect of operationalization, we consider it to be out of scope as it is highly context dependent.

\paragraph{\textbf{Group Data Requirements (How)}}
\begin{itemize}
    \item \textit{What is it}: We propose that this axis ranges from group data availability in (in order of most to least preferable) - Nowhere, validation data, training \& validation data, training \& validation \& inference data.
    \item \textit{Why it matters}: The collection of group data itself can be a point of contention. As \citet{AndrusSpitzer, bent2019algorithmic, xiang2020reconciling} point out, the legality of collecting and using group data in models is ambiguous from a discrimination perspective, even if only for auditing purposes. On top of the legal risks, \citet{AndrusDemographic} argue that group data collection even for honest purposes of inducing fairness can result in harmful social effects. On the other hand, \citet{BogenAwareness, vzliobaite2016using} present cases where group data is necessary and beneficial for algorithms to be fair. Overall, this is to say that there does not seem to be a one-size-fits-all ruling for using group data.

    This motivates us to move beyond the dichotomous view of viewing group data as fully available or not. We consider the graduated scale of group data, which enables more flexible and complex fairness intervention algorithms to be used. As a concrete example, obtaining a small set (validation set) of group data from volunteers may enable practitioners to tune models for fairness. If the data is large enough, it may even permit training a model with group data that does not require access at inference time. On the extreme end, having full access to group data at training and inference would enable using a variety of post-processors. As \citet{AndrusSpitzer,grgic2018beyond} discuss, these different regimes can also have a graduated on public perception. The matter of perception is further nuanced by the fact that access to group data may be restricted select practitioners within an organization \cite{nandy_2021}. We therefore recommend practitioners to quantitatively study how far they can get under the different data availability regimes to justify the use of group data. We specifically demonstrate this process in our case study of predictive parity in Section \ref{section: Case Study}.

\end{itemize}

\paragraph{\textbf{Place of Intervention and Engineering Effort (Where)}}
\begin{itemize}
    \item \textit{What is it}: We propose that the current categorization of pre-/in-/post- processing is a clear stratification of where the intervention occurs and that each carries its pros and cons in practice with respect to engineering effort and scalability.
    \item \textit{Why it matters}: First and foremost, the engineering effort can vary widely between the three levels.  Post-processing interventions typically allow lightweight iteration, and AB testing between existing (base) models versus the base model with the post-processor attached, as the model can effectively be treated as an opaque box. For example, a post-processing tool for calibration (e.g. isotonic regression) has relatively few parameters to train and only requires the model predictions and true label.

    On the other hand, in-processing mitigation tends to require a deeper understanding of the base model architecture (see, for examples, the recent methodological survey \cite{HortMitigationSurvey}). For instance, some regularizers that work for one model may not work for another due to dependence on assumptions such as convexity or differentiability. Additionally, more engineering effort may be required to push customized changes to training schemes, loss functions, or tune the associated hyperparameters, which lengthens time to experimentation.

    The effort is further complicated when moving to pre-processing schemes, which tend to require a very fine-grained understanding of model details and features. In this regime, there are very fundamentally difficult factors to consider here as opposed to in-/post- processing; for instance, consider the distinction between numeric, categorical, and embedding variables. While understanding the bias of features is undeniably important \cite{grgic2018beyond}, it is effort-intensive, requires ad-hoc qualitative judgment, and is difficult to scale across organizations.

    The aforementioned engineering difficulties compound as many large-scale software companies opt  for a centralized entity to handle responsible/fair/ethical AI affairs (hereafter, "responsible AI teams")~\cite{kramer_2021, logan_2022, kloumann_tannen_2021, croak_gennai_2023}. A centralized model has the significant benefit of consolidating standard responsible AI principles and developing cohesive tooling. However, its main drawback is that highly customized solutions are difficult to implement and scale across the company.

    For transparency purposes, we find from the fairness intervention survey \cite{HortMitigationSurvey} that post-processing frameworks tend to enable finer control and better explainability than in-processing. For instance, whereas it is difficult to understand how strong in-processing regularizers need to be to achieve a certain effect, post-processing algorithms tend to resemble constrained optimization problems that adjust the base models' scores with interpretable parameters. Having this transparency would largely aid in explaining the intervention to various stakeholders (e.g., legal and product teams, as well as users).

    To summarize, the qualitative consideration of where the intervention occurs poses significant practical consequences. Both researchers and practitioners must think ahead on the question of \textit{where} for the sake of engineering effort/scalability and transparency, tailoring to the idiosyncracies of each organization.
\end{itemize}

\paragraph{\textbf{Intervention Effectiveness}}
\begin{itemize}
    \item \textit{What is it}: Intervention effectiveness refers to how well a chosen methodology achieves its fairness goal. For instance, if pursing an equalized odds type of fairness, one may look at the largest largest differences in TPR/FPRs across the groups. An important distinction here is also for whom the intervention should work for (e.g. if pursuing group granularity, individual fairness). In our case study of predictive parity fairness, we will measure the worst group calibration error.
    \item \textit{Why it matters}: This is the typical end-goal of the fairness practitioner. In measuring effectiveness, we encourage practitioners and researchers to do so with respect to the above two points. A post-processing algorithm that requires group data at inference may yield better fairness effects than an in-processing algorithm. However, whether the improvement is worth the extra data is an empirical problem that we believe deserves benchmarking. The methodological survey \cite{HortMitigationSurvey} indicates that most benchmarks are done with respect to the same category of interventions (in-/pre-/post-). However, some are done across categories and it is not clear if such benchmarks are consistent in the amount of group data being used. We therefore encourage that given the practical importance of data requirements, benchmarks in academic and industrial settings be done with respect to both intervention category and the data requirements.
\end{itemize}

\paragraph{\textbf{Model Performance}}
\begin{itemize}
    \item \textit{What is it}: Model performance refers to the difference in model performance compared to the baseline when applying a fairness intervention. This may typically be measured by a performance metric such as accuracy, but other times may involve downstream metrics in products such as click-through-rate.
    \item \textit{Why it matters}: Demonstrating an adequate level of model performance post-intervention is typically a requirement to get buy-in from product teams and other stakeholders. Although some recent works have shown empirically that the fairness versus performance trade-off is not as steep as one may assume \cite{Rodolfa_2021}, it still remains a valid concern. It appears that this is a well-known factor and we encourage practioners to continue quantitatively benchmarking fairness interventions on the basis of model performance.
\end{itemize}

This framework overall aims to capture the major considerations in operationalizing fairness that are quantifiable and enable benchmarking to some extent, as we believe that this helps practitioners decide how to make trade-offs between the pillars. We remark that while these are the classical categorizations in ML pipelines, there are still applications that use group data in ways that fall outside of these categories (for instance at the data collection step \cite{caiAdaptiveSampling, li2022more}, we also propose one such method in the next section at the feature selection step). These methods should be considered, but our work focuses on bringing more structure and order to the majority of fairness intervention work in the highlighted categories \cite{HortMitigationSurvey}. With this four pillar framework in mind, we pivot our work to demonstrating this on a case study of predictive parity fairness. In our study, we will focus on quantitative demonstrations of these trade-offs in order to make a decision.
\section{Intro}
\subsection{Holistically Approaching Algorithmic Fairness}
As machine learning (ML) models increasingly play a role in day-to-day life, algorithmic fairness has been steadily gaining attention in both public and academic eyes. Fortunately, there has been no shortage of research into measuring and achieving different definitions of algorithmic fairness, with a recent set of comprehensive surveys (\cite{ChenTestingSurvey}, \cite{HortMitigationSurvey}) describing and categorizing over 300 fairness intervention\footnote{The authors of the survey call these methods "bias mitigation" methods and semantics vary across institutions. For this work, we will refer to all algorithms with the end goal of inducing some measure of fairness to be "fairness interventions" or just "interventions."} algorithms and 100 testing algorithms. However, alongside the spectrum of such algorithms comes the practical problem of selecting the "right" intervention strategy in real-world applications. Several recent works have recognized that this is a critical choice that stretches beyond sole consideration of the canonical fairness effectiveness versus model performance trade-off. We contribute to this literature by proposing a selection framework that emphasizes two additional key dimensions for evaluating fairness interventions. The first dimension is \textit{how} the algorithm is applied, specifically in terms of its protected/sensitive group data requirements\footnote{Examples include ethnicity, gender identity, sexual alignment, age, veteran status, etc. We hereafter refer to these general attributes as "group data."}, and the second dimension is with respect to \textit{where} the intervention is applied between pre-/in-/post-processing, which has significant implications for engineering effort in real settings. We identified these aspects as critical to operationalization as they have major implications on the public and legal perception of the fairness intervention as well as the engineering scalability of the solution.

To concretely motivate the problem, consider the commonly-used categorization of fairness intervention algorithms into three categories - pre-processing, in-processing, and post-processing, which respectively correspond to adjustments of the data, model, and model outputs. Post-processing solutions are effective in that they directly tackle the fairness problem at the decision level and are considered scalable from an engineering perspective as they can be readily detached from and/or applied onto models with little to no fine-tuning. However interventions at this level typically use the protected attribute (e.g., gender, race, religion) as part of the output adjustment (see \cite{Hardtetal_NIPS2016, pleiss2017fairness, NandyEOdds, Hsu2022} for examples). On the other hand, interventions at the in-processing stage may use the protected attribute in regularization fashion (e.g, \cite{Celis2019, cotterNonconvexConstrained, Rezaei_2020}). While these approaches may be harder to implement and tune across many models (and therefore arguably lack engineering scalability), the critical difference from an algorithmic perspective is that \emph{they do not require using the protected attribute at inference}. While this is a minor technical difference, it can carry significant ramifications from legal and public opinion perspectives \cite{AndrusDemographic, grgic2018beyond}. Hence, on top of the traditional trade-offs between model performance and fairness, we introduce two important axes to consider in the data availability and the engineering effort. We capture these ideas in Figure \ref{fig:Framework}.

\begin{center}
    \begin{figure}[h]
    \caption{Our proposed framework for operationalizing fairness. }
    \label{fig:Framework}
    \includegraphics[width=\textwidth]{Images/Framework.png}
    \end{figure}
\end{center}

\subsection{Previous Work and Contributions}
Several recent works have drawn attention to the complexities of fairness in ML beyond the algorithms. These works bring in interdisciplinary perspectives on fairness, including arguments from the legal \cite{Bringas_Colmenarejo_2022, bent2019algorithmic, xiang2020reconciling}, economic \cite{rambachan2020economic, AndrusDemographic}, moral \cite{Loi2022IsCA, Bringas_Colmenarejo_2022}, and industrial \cite{pfohl2021empirical, BogenAwareness, bakalarFairnessOnTheGround, AndrusSpitzer} perspectives. These works highlight the need to concretely evaluate how group data is used for fairness purposes and to what extent interventions work without such data. The essay by Andrus and Villeneuve \cite{AndrusDemographic} is of particular pertinence to our work, as it calls for practitioners to think twice before blindly pursuing data gathering at all costs due to the hazards ranging from organizational risks to individual hazards to the organization's customers/members.

Many of these works instill much-needed grounding perspectives at the higher system/organization level\footnote{By system level, we are referring to broad directives such as how group data can be collected from users, differences between human and algorithmic bias, who deserves the benefit from algorithmic fairness, etc. This is as opposed to granular decisions at the model or product level such as selecting a specific point of intervention or even a specific fairness intervention algorithm itself.}. While many of these works instill much-needed grounding perspective on fairness at the system level, there is now a need to bridge these system level directives with the growing arsenal of fairness intervention methods. Some examples of such investigations are \cite{vzliobaite2016using}, which theoretically argues for the use of group data during model training and demonstrates it on salary data. Another example is \cite{grgic2018beyond}, which introduces and addresses the problem of process fairness and demonstrates its on data sourced from Amazon's Mechanical Turk. The latter is particularly relevant for our work, as it analyzes the human perceived fairness of using individual features and solves the problem of finding a fair and good enough subset of features. Likewise, we emphasize in our work that fairness choices can and should be viewed as a spectrum rather than Boolean switches.

Following in these footsteps, our work makes two main contributions that tie the system level fairness discussions fairness down to quantitative evaluations. First, we propose a framework that highlights four key pillars for selecting a fairness intervention solution that holistically considers model performance and algorithmic effectiveness alongside two operationally critical axes - data availability and engineering effort. Beyond just providing insights and heuristics for navigating the trade-offs between these pillars in industrial settings, our work shows that these trade-offs can be evaluated across fairness intervention algorithms as we demonstrate in Figure \ref{fig:Aggregate ACS Frontier}. This leads into our second contribution as we demonstrate our framework in the context of predictive parity fairness and present novel methodological findings; we study close to 400 methodological variations across two major model types (XGBoost vs. Neural Net) and ten datasets. Methodological insights derived from our empirical study inform the practical design of ML workflow with fairness as a central concern. We find that predictive parity is difficult to achieve without using group data, and despite requiring group data during model training (but not inference), distributionally robust methods provide significant Pareto improvement. Moreover, a regular XGBoost model often Pareto-dominates neural networks with fairness interventions, highlighting the importance of model inductive bias. As collection of sensitive data becomes easier yet riskier, this study aims to illuminate the need to focus on a central path forward between the full or no allowance of sensitive attributes in ML pipelines.