Machine learned models exhibit bias, often because the datasets used to train them are biased. This presents a serious problem for the deployment of such technology, as the resulting models might perform poorly on populations that are minorities within the training set and ultimately present higher risks to them. We propose to use high-fidelity computer simulations to interrogate and diagnose biases within ML classifiers. We present a framework that leverages Bayesian parameter search to efficiently characterize the high dimensional feature space and more quickly identify weakness in performance. We apply our approach to an example domain, face detection, and show that it can be used to help identify demographic biases in commercial face application programming interfaces (APIs).
Summary:
=========
The paper uses a proof-of-concept Bayesian parameter search-based simulation in virtual environment to probe biases of an already trained model towards specific categories that may have been sparsely represented in the training set. Understanding bias in trained models, especially in models involving end-to-end deep neural networks learners, is of high importance in machine learning. More specifically, probing the source of unintentional bias introduced as a result of skewed distribution in the training set and dissecting the biased performance is important for many applications such as surveillance, criminal profiling, medical diagnosis and predicting creditworthiness of a person. 
The authors used four commercial face recognition APIs (by Microsoft, Google, IBM, and Face++) as test bed for this investigation.
Strength:
========
- The paper reads well and is easy to follow.
- The application of face detection and recognition is a good choice as it is precursor to detailed analysis in surveillance and criminal profiling.
- The choice of the controlled Bayesian parameter search enables one to control the amount of variation with respect to the expected uncertainty in performance of the classifier on the generated input from the simulator.
- The use of standardized measures such as Fitzpatrick’s skin tone and FACS intensity help in replication and consistency in the evaluation.
Weakness:
=========
- Although evaluating commercial APIs is good enough in demonstrating the existence of the bias, access to the trained model with possibility to retrain the model in such a way to mitigate a bias in particular parameter could have helped to further tie the drop in performance to the parameter variation.
- This work is preliminary and only involves a single person face manipulated in the parameter space. It lacks diversity of in samples and as such limits the analysis to draw strong conclusions. As such a generative network (such as a GAN trained to generate diverse samples while controlling for the parameters under investigation could have helped to draw more generalized conclusion.
- The age parameter variation is not convincingly different across the values considered. It would have been interesting to use models trained for age progression such as [1] for a more diverse variation in the age parameter space.
Minor comments:
===============
- Figures 4 and 5 could have used better captions describing the ranges for instance for age 1 is older and for skin tone 1 is darker (although indicated in Fig. 3). Captions should be self contained. Fig. 5 caption should describe the chance performance in each case.
- The manuscript should be revised to make in text citation formats consistent (some places it uses authors (year) and other places it uses (authors, year)). Also, minor punctuation and syntactic errors should be fixed.
[1] Yang, H., Huang, D., Wang, Y. and Jain, A.K., 2017. Learning face age progression: A pyramid architecture of gans. arXiv preprint arXiv:1711.10352.