Answers.Body: <p>There is the case of the tesla accident where the car was in autopilot and crashed into a truck because it appears the vehicle mistook a lightly coloured truck for the sky, killing the driver: <a href="https://www.newscientist.com/article/2095740-tesla-driver-dies-in-first-fatal-autonomous-car-crash-in-us/" rel="noreferrer">https://www.newscientist.com/article/2095740-tesla-driver-dies-in-first-fatal-autonomous-car-crash-in-us/</a></p>
  
  <p>Having said that, it appears the car had been trying to tell the driver to pay attention: <a href="http://uk.reuters.com/article/us-tesla-crash-idUKKBN19A2XC" rel="noreferrer">http://uk.reuters.com/article/us-tesla-crash-idUKKBN19A2XC</a> </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-17T21:26:40.780Z
Answers.LastActivityDate: 2017-08-17T21:26:40.780Z
Answers.UserDisplayName: redhqs
Answers.Body: <p>The infamous <a href="https://en.wikipedia.org/wiki/2010_Flash_Crash" rel="nofollow noreferrer">Flash Crash of 2010</a> may qualify. </p>
  
  <p>It didn't involve Artificial General Intelligence (which is still a hypothetical) or even "strong narrow AI" (such as AlphaGo) but does involve algorithmic decision-making, which is a form of basic Artificial Intelligence.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Algorithmic_trading" rel="nofollow noreferrer">Algorithmic trading</a> already represents a significant percentage of all market activity, and I suspect that percentage will only increase. </p>
  
  <p>From Business Insider: <a href="http://www.businessinsider.com/algos-could-trigger-stock-market-crash-2017-5" rel="nofollow noreferrer">Algos could trigger the next stock market crash</a> </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-08-17T19:34:44.720Z
Answers.LastActivityDate: 2017-08-17T19:34:44.720Z
Answers.UserDisplayName: DukeZhou
Body: <p>Are there any real-world examples of unintentional "bad" AI behaviour? I'm not looking for hypothetical arguments of malicious AI (AI in a box, paperclip maximizer), but for actual instances in history where some AI directly did something bad due to its direct action that was not intended behavior.</p>
  
  <p>Interpretations of the meaning of "AI", "bad", and "unintentional" are left open due to obvious reasons. Be free with your interpretation, but use some common sense please.</p>
  
  <p>Ex: Microsoft Tay became a racist not too long after being hooked up to the internet, thanks to internet trolls "teaching" her bad things.</p>
  
  <p>I can't think of any other instances. So the following examples are just hypothetical scenarios to demonstrate what I mean.</p>
  
  <p>Ex: A self-driving car drove off-track after being presented with an adversarial example, crashing into people.</p>
  
  <p>Ex: Surgery bot goes haywire and accidentally kills someone.</p>
  
  <p>Ex: Weaponized drone targets civilians against its design.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-13T18:37:32.673Z
Comments.Text: Define **bad**.
Comments.UserDisplayName: Rahul
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-13T19:03:16.073Z
Comments.Text: question edited to address your problem.
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-13T19:05:41.710Z
Comments.Text: And that makes it even more broad. Does "bad" means unemployment or AI enslaving humans ? What exactly is it ?
Comments.UserDisplayName: Rahul
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-13T19:20:59.980Z
Comments.Text: I dunno. The examples you provide suggest you have an idea of what to expect, so do you have any examples of adversarial AI behaviour in the wild under those interpretations? "Be free with your interpretation, but use common sense please."
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-14T22:36:36.060Z
Comments.Text: By "in the wild", you mean fully independent automata, not directed by humans?  (i.e. partisan, adversarial investment algorithms are not "in the wild") I think the use of the idiom is good, but just wanted to make sure I'm understanding your meaning.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-15T14:21:22.750Z
Comments.Text: question edited to address your problem. more specifically addressed to you: i'm just looking for instances that has "actually happened before", as opposed to being hypothetical (ai in box)
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-17T12:10:08.917Z
Comments.Text: I believe any example of overturned [AI sentence in court](https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/) would count. I don't know of any - but I'm quite scared to think how many innocent people may be in prison, because the AI judged basing on ethnicity, social background, education level and other "risk assessment" factors, while downplaying facts of that specific case. Knowing such algorithms -, if 30 statistical factors say "he's the type of person that would do it" and 1 individual fact says "he didn't do it - he wasn't there", the AI will say "guilty."
Comments.UserDisplayName: SF.
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-08-17T17:57:59.573Z
Comments.Text: the algorithm in question doesn't sentence or give verdict in court. it provides a metric for risk assessment, which judges then use when sentencing people. although strictly speaking this example fits the criteria above, its not really what i'm looking for. perhaps you also think this way even just a little, from the observation that you chose to comment here rather than post a full fledged answer.
Comments.UserDisplayName: k.c. sayz 'k.c sayz'
Tags: <ai-safety>
Title: Concrete examples of unintentional adversarial AI behaviour
UserDisplayName: k.c. sayz 'k.c sayz'
ThreadText: # Concrete examples of unintentional adversarial AI behaviour
  
  (Asked by: username_0 on 2017-08-13)
  
  Are there any real-world examples of unintentional "bad" AI behaviour? I'm not looking for hypothetical arguments of malicious AI (AI in a box, paperclip maximizer), but for actual instances in history where some AI directly did something bad due to its direct action that was not intended behavior.
  
  Interpretations of the meaning of "AI", "bad", and "unintentional" are left open due to obvious reasons. Be free with your interpretation, but use some common sense please.
  
  Ex: Microsoft Tay became a racist not too long after being hooked up to the internet, thanks to internet trolls "teaching" her bad things.
  
  I can't think of any other instances. So the following examples are just hypothetical scenarios to demonstrate what I mean.
  
  Ex: A self-driving car drove off-track after being presented with an adversarial example, crashing into people.
  
  Ex: Surgery bot goes haywire and accidentally kills someone.
  
  Ex: Weaponized drone targets civilians against its design.
  
  **Tags:** ai-safety
  
  > **Comments:**
  > **username_1**: Define **bad**.  
  > **username_0**: question edited to address your problem.  
  > **username_1**: And that makes it even more broad. Does "bad" means unemployment or AI enslaving humans ? What exactly is it ?  
  > **username_0**: I dunno. The examples you provide suggest you have an idea of what to expect, so do you have any examples of adversarial AI behaviour in the wild under those interpretations? "Be free with your interpretation, but use common sense please."  
  > **username_2**: By "in the wild", you mean fully independent automata, not directed by humans?  (i.e. partisan, adversarial investment algorithms are not "in the wild") I think the use of the idiom is good, but just wanted to make sure I'm understanding your meaning.  
  > **username_0**: question edited to address your problem. more specifically addressed to you: i'm just looking for instances that has "actually happened before", as opposed to being hypothetical (ai in box)  
  > **username_3**: I believe any example of overturned [AI sentence in court](https://www.wired.com/2017/04/courts-using-ai-sentence-criminals-must-stop-now/) would count. I don't know of any - but I'm quite scared to think how many innocent people may be in prison, because the AI judged basing on ethnicity, social background, education level and other "risk assessment" factors, while downplaying facts of that specific case. Knowing such algorithms -, if 30 statistical factors say "he's the type of person that would do it" and 1 individual fact says "he didn't do it - he wasn't there", the AI will say "guilty."  
  > **username_0**: the algorithm in question doesn't sentence or give verdict in court. it provides a metric for risk assessment, which judges then use when sentencing people. although strictly speaking this example fits the criteria above, its not really what i'm looking for. perhaps you also think this way even just a little, from the observation that you chose to comment here rather than post a full fledged answer.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2017-08-17. Score: 5
  
  There is the case of the tesla accident where the car was in autopilot and crashed into a truck because it appears the vehicle mistook a lightly coloured truck for the sky, killing the driver: <https://www.newscientist.com/article/2095740-tesla-driver-dies-in-first-fatal-autonomous-car-crash-in-us/>
  
  Having said that, it appears the car had been trying to tell the driver to pay attention: <http://uk.reuters.com/article/us-tesla-crash-idUKKBN19A2XC>
  
  
  ---
  
  ### Answer by username_2 on 2017-08-17. Score: 4
  
  The infamous [Flash Crash of 2010](https://en.wikipedia.org/wiki/2010_Flash_Crash) may qualify.
  
  It didn't involve Artificial General Intelligence (which is still a hypothetical) or even "strong narrow AI" (such as AlphaGo) but does involve algorithmic decision-making, which is a form of basic Artificial Intelligence.
  
  [Algorithmic trading](https://en.wikipedia.org/wiki/Algorithmic_trading) already represents a significant percentage of all market activity, and I suspect that percentage will only increase.
  
  From Business Insider: [Algos could trigger the next stock market crash](http://www.businessinsider.com/algos-could-trigger-stock-market-crash-2017-5)
  
  
  ---
  
  