They likely  didn&#x27;t purposefully &quot;do&quot; anything wrong -- at least in the sense of some racist engineer tampering with the system to have it come out with those results.<p>But this is the nature of machine learning algorithms, including both the process of supervision and ability to view the impact feedbacks have on the algorithms, and also, the impact the quality of the training set given to the algorithms. At a lesser company, the problem could be as simple as very few black people represented in the training set, so that when the algorithm sees a dark-colored human-like shape, it is more &quot;likely&quot; that that shape is a gorilla  (which is human like and pretty much always has dark fur) than it is a human, because the algorithm was trained mostly on light-colored humans. The Google Photos algorithm obviously takes in more kinds of input and factors besides visual composition so there was probably more to it than this.<p>Or maybe not...who knows? I&#x27;m not interested in reviving a discussion about importance of diversity in the engineering workforce, but this is one kind of problem that can slip by the most competent and well-intentioned of engineers simply because they&#x27;re less aware of how disenfranchisement can propagate into technical problems, no matter how correct and powerful the math behind the algorithm.<p>Another example from a few years back was when HP released a auto-tracking webcam that became infamous after a black retail employee uploaded a YouTube video of how the camera ignored him but not his white co-worker:<p><a href="http:&#x2F;&#x2F;www.cnn.com&#x2F;2009&#x2F;TECH&#x2F;12&#x2F;22&#x2F;hp.webcams&#x2F;" rel="nofollow">http:&#x2F;&#x2F;www.cnn.com&#x2F;2009&#x2F;TECH&#x2F;12&#x2F;22&#x2F;hp.webcams&#x2F;</a><p>I&#x27;m in 100% agreement that this was likely not HP&#x27;s intentional fault, and also that face detection of darker complexions is computationally more complex than it is for lighter complexions because of how contrast is used by the algorithm...but I most definitely know that if I were an HP engineer, and if the CEO and&#x2F;or my direct boss were black and tried out a prototype that behaved as it does in the aforementioned YouTube video, there is almost no fucking way that the product would be released as-is, with my excuse being &quot;Well, accurately detecting black faces requires a much more complicated training set -- that&#x27;s just how math works!&quot;