Answers.Body: <p>A quite significant issue is where some AI systems have mislabeled black people as being gorillas (I suspect a major cause is the training data being insufficiently diverse, balanced and representative). Two examples are:</p>
  <ul>
  <li>In 2015, this occurred with Google's Photo App (e.g., see <a href="https://www.cnet.com/tech/services-and-software/google-apologizes-for-algorithm-mistakenly-calling-black-people-gorillas/" rel="nofollow noreferrer">Google apologizes for algorithm mistakenly calling black people 'gorillas'</a>). Note that, for at least 3 years, Google apparently &quot;fixed&quot; their system just by removing gorillas as being an option, with this explained in <a href="https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai" rel="nofollow noreferrer">Google ‘fixed’ its racist algorithm by removing gorillas from its image-labeling tech</a>.</li>
  <li>Facebook's AI recommendation system had a similar problem (e.g., see <a href="https://www.bbc.com/news/technology-58462511" rel="nofollow noreferrer">Facebook apology as AI labels black men 'primates'</a>).</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-20T06:37:34.050Z
Answers.LastActivityDate: 2022-09-20T15:43:28.450Z
Answers.LastEditDate: 2022-09-20T15:43:28.450Z
Answers.UserDisplayName: John Omielan
Answers.Body: <p>A couple of examples could be:</p>
  <ul>
  <li><p><em>Image classifiers learning different properties than the actual target</em>: many books reference the case of a perceptron trained on detecting tanks which learned to actually predict good or bad weather in the background, ignoring completely the tanks. First cited in <a href="https://www.gwern.net/docs/www/www.jefftk.com/d21a41fba82b2fbb43d273b1847722f2b18eb387.pdf" rel="nofollow noreferrer">What Artificial Experts Can and Cannot Do</a>.
  (notably this is most likely a <a href="https://www.gwern.net/Tanks#origin" rel="nofollow noreferrer">urban legend</a>, but still a very much realistic situation that anybody working in computer vision will face soon or later).</p>
  </li>
  <li><p><em>Amazon recruitment algorithm biased towards men</em>: this is totally real and it has been analyzed in several paper, I'll just link the first one I found <a href="https://www.researchgate.net/publication/331967358_Encoded_Bias_in_Recruitment_Algorithms" rel="nofollow noreferrer">Encoded Bias in Recruitment Algorithms</a>. Again another case which remember us that machine learning and AI in general are data driven. An algorithm will learn and always be limited by what's in the data, including stereotypes and prejudices in case of natural language processing.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-09-19T08:51:52.070Z
Answers.LastActivityDate: 2022-09-19T08:51:52.070Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>For a short presentation about AI, I am looking for examples where AI failed and thereby shows the limits of itself.</p>
  <p>I remember there was one example,where an image classifier was given an image of pink animals (I think sheep) on a tree and classified it as &quot;Birds on a tree&quot;. I think this example showed what AI might do if the given example is not represented in the training data.</p>
  <p>But I cannot find that example anymore (and I need a source).</p>
  <p>Anyone knows of examples, which are documented, that I could give and show the problem in a similar way?</p>
  
Tags: <computer-vision><reference-request><image-recognition><social>
Title: Examples where AI fails in revealing ways
UserDisplayName: Nathan
ThreadText: # Examples where AI fails in revealing ways
  
  (Asked by: username_0 on 2022-09-19)
  
  For a short presentation about AI, I am looking for examples where AI failed and thereby shows the limits of itself.
  
  I remember there was one example,where an image classifier was given an image of pink animals (I think sheep) on a tree and classified it as "Birds on a tree". I think this example showed what AI might do if the given example is not represented in the training data.
  
  But I cannot find that example anymore (and I need a source).
  
  Anyone knows of examples, which are documented, that I could give and show the problem in a similar way?
  
  **Tags:** computer-vision, reference-request, image-recognition, social
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2022-09-20. Score: 1 | Accepted answer
  
  A quite significant issue is where some AI systems have mislabeled black people as being gorillas (I suspect a major cause is the training data being insufficiently diverse, balanced and representative). Two examples are:
  
  * In 2015, this occurred with Google's Photo App (e.g., see [Google apologizes for algorithm mistakenly calling black people 'gorillas'](https://www.cnet.com/tech/services-and-software/google-apologizes-for-algorithm-mistakenly-calling-black-people-gorillas/)). Note that, for at least 3 years, Google apparently "fixed" their system just by removing gorillas as being an option, with this explained in [Google ‘fixed’ its racist algorithm by removing gorillas from its image-labeling tech](https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai).
  * Facebook's AI recommendation system had a similar problem (e.g., see [Facebook apology as AI labels black men 'primates'](https://www.bbc.com/news/technology-58462511)).
  
  
  ---
  
  ### Answer by username_2 on 2022-09-19. Score: 3
  
  A couple of examples could be:
  
  * *Image classifiers learning different properties than the actual target*: many books reference the case of a perceptron trained on detecting tanks which learned to actually predict good or bad weather in the background, ignoring completely the tanks. First cited in [What Artificial Experts Can and Cannot Do](https://www.gwern.net/docs/www/www.jefftk.com/d21a41fba82b2fbb43d273b1847722f2b18eb387.pdf).
    (notably this is most likely a [urban legend](https://www.gwern.net/Tanks#origin), but still a very much realistic situation that anybody working in computer vision will face soon or later).
  * *Amazon recruitment algorithm biased towards men*: this is totally real and it has been analyzed in several paper, I'll just link the first one I found [Encoded Bias in Recruitment Algorithms](https://www.researchgate.net/publication/331967358_Encoded_Bias_in_Recruitment_Algorithms). Again another case which remember us that machine learning and AI in general are data driven. An algorithm will learn and always be limited by what's in the data, including stereotypes and prejudices in case of natural language processing.
  
  
  ---
  
  