Counterfactual Fairness
Machine learning can impact people with legal or ethical consequences when it is used to automate decisions in areas such as insurance, lending, hiring, and predictive policing. In many of these scenarios, previous decisions have been made that are unfairly biased against certain subpopulations, for example those of a particular race, gender, or sexual orientation. Since this past data may be biased, machine learning predictors must account for this to avoid perpetuating or creating discriminatory practices. In this paper, we develop a framework for modeling fairness using tools from causal inference. Our definition of counterfactual fairness captures the intuition that a decision is fair towards an individual if it the same in (a) the actual world and (b) a counterfactual world where the individual belonged to a different demographic group. We demonstrate our framework on a real-world problem of fair prediction of success in law school.
This paper presents an interesting and valuable contribution to the small but growing literature on fairness in machine learning.  Specifically, it provides at least three contributions: (1) a definition of counter factual fairness; (2) an algorithm for learning a model under counter factual fairness; and (3) experiments with that algorithm. 
The value and convincingness of each of these contributions declines steadily. The value of the contributions of the current paper is sufficient for acceptance, though significant improvements could be made in the clarity of exposition of the algorithm and the extent of the experimentation with the algorithm.
Section 4.2 outlines why is is likely that very strong assumptions will need to be made to effectively estimate a model of Y under counterfactual fairness.  The assumptions (and the implied analysis techniques) suggest conclusions that will not be particularly robust to violations of those assumptions.  This implies a need for significant empirical evaluation of what happens when those assumptions are violated or when other likely non-optimal conditions hold.  Realistic conditions that might cause particular problems include measurement error and non-homogeneity of causal effect.  Both of these are very likely to be present in the situations in which fairness is of concern (for example, crime, housing, loan approval, and admissions decisions).  The paper would be significantly improved by experiments examining model accuracy and fairness under these conditions (and under errors in assumptions).
The discussion of fairness and causal analysis omits at least one troubling aspect of real-world discrimination, which is the effect of continuing discrimination after fair decisions are made. For example, consider gender discrimination in business loans.  If discriminatory actions of customers will reduce the probability of success for a female-owned business, then such businesses will have a higher probability of default, even if the initial loan decisions are made in an unbiased way. It would be interesting to note whether a causal approach would allow consideration of such factors and (if so) how they would be considered.
Unmeasured selection bias would seem to be a serious threat to validity here, but is not considered in the discussion.  Specifically, data collection bias can affect the observed dependencies (and, thus, the estimation of potential unmeasured confounders U and the strength of causal effects).  Selection bias can occur due to selective data collection, retention, or reporting, when such selection is due to two or more variables.  It is relatively easy to imagine selection bias arising in cases where discrimination and other forms of bias are present.  The paper would be improved by discussing the potential effects of selection bias.
A few minor issues: The paper (lines 164-166 and Figure 1) references Pearl's twin network, but the explanation of twin networks in caption of Figure 1 provides insufficient background for readers to understand what twin networks are and how they can be interpreted.  Line 60 mentions "a wealth of work" two sentences after using "wealth" to refer to personal finances, and this may be confusing to readers.