Unexpected sources of bias in artificial intelligence
I think we need to stop using the word &quot;bias&quot; and &quot;prejudice&quot;. The term &quot;bias&quot; has a real, well-defined meaning in statistics, and both of these terms imply <i>incorrect</i> decisions are being systematically made.<p>Only one example in this article is actually about incorrect decisions. Most examples are about unbiased learning systems doing exactly what they are designed to do, but people other than the designers wishing they did different things.<p>For example, consider &quot;conflicting goals bias&quot;. This isn&#x27;t a bias - the system is maximizing clicks exactly as desired. It&#x27;s just that some random third party wishes the system were actually trying to mitigate a nonexistent psychological problem (namely stereotype threat) instead [1].<p>What they call &quot;similarity bias&quot; is the same thing. The system is attempting to show people stories they like (and probably does a good job at it), but the author wishes instead that the selection of stories was closer to what a journalist might choose.<p>Another social &quot;bias&quot; - namely &quot;redlining&quot;&#x2F;redundant encoding&#x2F;etc is actually the <i>elimination</i> of statistical bias. A lot of input metrics - e.g. SAT score, FICO score, etc - are biased in favor of non-Asian minorities (and against Asians) [2] and machine learning algorithms designed to find hidden patterns discover this bias and fix it.<p>Conflating &quot;someone is doing X but I wish they did Y&quot; with bias is not useful. It&#x27;s also not useful to conflate the elimination of socially desirable biases with introducing new biases.<p>[1] Stereotype threat has repeatedly failed to replicate. Funnel plots suggest it only existed to begin with due to publication bias. <a href="http:&#x2F;&#x2F;www.sciencedirect.com.sci-hub.cc&#x2F;science&#x2F;article&#x2F;pii&#x2F;S0022440514000831" rel="nofollow">http:&#x2F;&#x2F;www.sciencedirect.com.sci-hub.cc&#x2F;science&#x2F;article&#x2F;pii&#x2F;...</a> <a href="https:&#x2F;&#x2F;dl.dropboxusercontent.com&#x2F;u&#x2F;85192141&#x2F;2013-ganley.pdf" rel="nofollow">https:&#x2F;&#x2F;dl.dropboxusercontent.com&#x2F;u&#x2F;85192141&#x2F;2013-ganley.pdf</a> <a href="https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Stereotype_threat#Failures_to_replicate_and_publication_bias" rel="nofollow">https:&#x2F;&#x2F;en.wikipedia.org&#x2F;wiki&#x2F;Stereotype_threat#Failures_to_...</a> <a href="https:&#x2F;&#x2F;replicationindex.wordpress.com&#x2F;tag&#x2F;stereotype-threat-and-womens-math-performance&#x2F;" rel="nofollow">https:&#x2F;&#x2F;replicationindex.wordpress.com&#x2F;tag&#x2F;stereotype-threat...</a><p>[2] See for example Figure 7 here: <a href="https:&#x2F;&#x2F;drive.google.com&#x2F;file&#x2F;d&#x2F;0B-wQVEjH9yuhanpyQjUwQS1JOTQ&#x2F;view" rel="nofollow">https:&#x2F;&#x2F;drive.google.com&#x2F;file&#x2F;d&#x2F;0B-wQVEjH9yuhanpyQjUwQS1JOTQ...</a>