Answers.Body: <p>As far as I know, Tesla cars autopilot is not a 100% AI pilot, it's an <em>assitant</em>: as it detects hands off wheel it slows down, so it's incorrect to speak about AI mistake: it is not trained/designed to drive a car all by itself. A human driver is responsible in that incident.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-20T15:07:40.497Z
Answers.Text: How does this answer the actual question?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-09T19:59:30.093Z
Answers.LastActivityDate: 2016-08-09T19:59:30.093Z
Answers.UserDisplayName: Alexey Vesnin
Answers.Body: <p>Tesla model S has <a href="https://www.tesla.com/models" rel="nofollow noreferrer">Autopilot</a> which allows to steer within a lane, change lanes with the simple tap of a turn signal, and can manage speed by using traffic-aware cruise control. Multiple digital controls helps to avoid collisions. Based on that, this isn't fully self-driving car.</p>
  <p>However it is using a computer vision detection system, but it is not intended to be used hands-free.</p>
  <p>So basically what is known is that the accident involved the side of a truck trailer (of a large white 18-wheel truck) and most likely the camera had a washed out of picture possibly due to glare or blooming from overexposure which made that the side of the trailer white and thin which failed to distinguish with the sky which was bright as well.</p>
  <p>This may have happened in part, because the crash-avoidance system only engage when both radar and vision system detect an obstacle which could not happen.</p>
  <p>Further more it was suggested by <em>The Associated Press</em> that the driver most likely was watching a <em>Harry Potter</em> at the time of the crash and assuming system would alert Brown, we don't know if he was able to retake controls quickly enough to avoid impact. As mentioned again, the system wasn't intended for hands-free driving and parts of the system was unfinished. Not to mention that the car was driving with full speed under the trailer.</p>
  <p>Tesla officially said about this crash in a statement on its website:</p>
  <blockquote>
  <p><strong>The high ride height of the trailer combined with its positioning across the road and the extremely rare circumstances</strong> of the impact caused the Model S to pass under the trailer, with the bottom of the trailer impacting the windshield of the Model S.</p>
  <p>Neither Autopilot nor the driver noticed the white side of the tractor-trailer against a brightly lit sky, so the brake was not applied.</p>
  </blockquote>
  <p>They also said, according to techno-optimists, that they will tweaks their code, so this particular case won't happen again.</p>
  <p>To summarize, this was a 'technical failure' of braking system and most likely Autopilot was not at as Tesla told Senate.</p>
  <p><a href="https://i.sstatic.net/obdLM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/obdLM.png" alt="The New York Times |Source: Florida traffic crash report" /></a></p>
  <p><sup>The New York Times |Source: Florida traffic crash report</sup></p>
  <p>Sources:</p>
  <ul>
  <li><a href="http://www.freep.com/story/money/cars/2016/07/01/tesla-autopilot-death-highlights-autonomous-risks/86591130/" rel="nofollow noreferrer">Tesla Autopilot death highlights autonomous risks</a></li>
  <li><a href="https://robotfuturesbook.wordpress.com/2016/07/01/layers-of-autonomy" rel="nofollow noreferrer">Layers of Autonomy</a></li>
  <li><a href="http://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html" rel="nofollow noreferrer">Inside the Self-Driving Tesla Fatal Accident</a></li>
  <li><a href="https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk" rel="nofollow noreferrer">Tesla driver dies in first fatal crash while using autopilot mode</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-12T03:22:24.100Z
Answers.LastActivityDate: 2016-08-12T03:38:43.983Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: kenorb
Answers.Body: <p>Tesla's technology is assistive, as Alexey points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies.</p>
  
  <hr>
  
  <p><em>Warning: I cannot find again the source critical to the next paragraph, and reading again pages over pages, I cannot find similar argument in other reports. I still remember vividly the point below, but please keep in mind it may be incorrect. The rest of the answer is weakly related, so I leave it all, with this warning.</em></p>
  
  <p>An independent report (link needed, I can't find it...) explained that the assistive system was unable to detect the truck due to an exceptionally low contrast (bright sky perceived as white---colour of the truck). The report also said that a human driver would have been unable to make the difference either. In other words, it is possible that car sensors (presumably camera) and the human eye could not have detected an obstacle, and could not have triggered any safety measure. This <a href="http://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html" rel="nofollow">short graphical explanation</a> sums up the car sensors: Camera, radar, GPS, etc.</p>
  
  <p>The assistive sub-system is based on proprietary AI technologies. We can <em>only speculate</em> under some hypothesis. _This is not very useful, honestly, except for <strong>illustration purpose</strong>. Assuming the assistive system relies on ML technologies to learn about obstacles from a video stream (such systems do exist):</p>
  
  <ul>
  <li>It may be that the learning data was not "good enough" to cover the truck scenario.</li>
  <li>It may be the technology was not powerful enough yet (lack generalization power, or simply too slow).</li>
  <li>It may be a hardware problem, notably from the sensors: If the "car's eyes" are defective, the "car's brain" (the assistive system) is unable to react properly.</li>
  </ul>
  
  <p>Why those technologies did not work in that case will remain a secret. We can say however that <em>any system</em>---whether built with AI technology or not---has limits. Beyond these limits, the system reaction is unpredictable: It could stop, reset, shutdown. The difficulty here is to define what a "default behaviour" is. A machine will basically do whatever it is designed to do, so an AI-based system too.</p>
  
  <p>We could speculate even more on what would happen if the assistive system was really autonomous, the elusive AGI, but that is really not the case here.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-10T11:10:21.850Z
Answers.Text: "The report also said that a human driver would have been unable to make the difference either. "  I haven't read the report, so I could be wrong, but I'm guessing that this wouldn't be true over a *sequence* of frames: there could be subtle cues that an object is present that a human could pick up on, but that image processing would likely dismiss as noise...
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-14T22:34:22.020Z
Answers.Text: Sorry, as I still cannot find the source... The sequence argument is speculative on this case, though. So many things can go wrong with such "early-stage" technology (Who would go to the Moon riding Apollo 11's hardware *today*?). Yes, it may be their algorithm at the time, lack of data, or the hardware is just not enough.
Answers.UserDisplayName: Eric Platon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-10T10:02:41.383Z
Answers.LastActivityDate: 2016-08-14T23:54:03.540Z
Answers.LastEditDate: 2016-08-14T23:54:03.540Z
Answers.UserDisplayName: Eric Platon
Body: <p>Do we know why Tesla's autopilot mistaken empty sky with a high-sided lorry which resulted in fatal crash involving a car in self-drive mode? Was it AI fault or something else? Is there any technical explanation behind this why this happened?</p>
  
  <p>The articles <a href="http://news.sky.com/story/tesla-driver-in-first-self-drive-fatal-crash-10330121" rel="nofollow noreferrer">Tesla Driver In First Self-Drive Fatal Crash</a> and <a href="http://www.theverge.com/2016/6/30/12072408/tesla-autopilot-car-crash-death-autonomous-model-s" rel="nofollow noreferrer">Tesla driver killed in crash with Autopilot active, NHTSA investigating</a> talk about this topic, but do not go in detail.</p>
  
Tags: <autonomous-vehicles><ai-security><ai-safety>
Title: Why did a Tesla car mistake a truck with a bright sky?
UserDisplayName: kenorb
ThreadText: # Why did a Tesla car mistake a truck with a bright sky?
  
  (Asked by: username_0 on 2016-08-09)
  
  Do we know why Tesla's autopilot mistaken empty sky with a high-sided lorry which resulted in fatal crash involving a car in self-drive mode? Was it AI fault or something else? Is there any technical explanation behind this why this happened?
  
  The articles [Tesla Driver In First Self-Drive Fatal Crash](http://news.sky.com/story/tesla-driver-in-first-self-drive-fatal-crash-10330121) and [Tesla driver killed in crash with Autopilot active, NHTSA investigating](http://www.theverge.com/2016/6/30/12072408/tesla-autopilot-car-crash-death-autonomous-model-s) talk about this topic, but do not go in detail.
  
  **Tags:** autonomous-vehicles, ai-security, ai-safety
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2016-08-10. Score: 4 | Accepted answer
  
  Tesla's technology is assistive, as username_1 points out, so this is not a case of an autonomous system (e.g. an AGI) doing some fatal stunt (the product name AutoPilot is famously misleading). Now on why the car assistance led to this tragic accident, there is some information related to AI technologies.
  
  ---
  
  *Warning: I cannot find again the source critical to the next paragraph, and reading again pages over pages, I cannot find similar argument in other reports. I still remember vividly the point below, but please keep in mind it may be incorrect. The rest of the answer is weakly related, so I leave it all, with this warning.*
  
  An independent report (link needed, I can't find it...) explained that the assistive system was unable to detect the truck due to an exceptionally low contrast (bright sky perceived as white---colour of the truck). The report also said that a human driver would have been unable to make the difference either. In other words, it is possible that car sensors (presumably camera) and the human eye could not have detected an obstacle, and could not have triggered any safety measure. This [short graphical explanation](http://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html) sums up the car sensors: Camera, radar, GPS, etc.
  
  The assistive sub-system is based on proprietary AI technologies. We can *only speculate* under some hypothesis. \_This is not very useful, honestly, except for **illustration purpose**. Assuming the assistive system relies on ML technologies to learn about obstacles from a video stream (such systems do exist):
  
  * It may be that the learning data was not "good enough" to cover the truck scenario.
  * It may be the technology was not powerful enough yet (lack generalization power, or simply too slow).
  * It may be a hardware problem, notably from the sensors: If the "car's eyes" are defective, the "car's brain" (the assistive system) is unable to react properly.
  
  Why those technologies did not work in that case will remain a secret. We can say however that *any system*---whether built with AI technology or not---has limits. Beyond these limits, the system reaction is unpredictable: It could stop, reset, shutdown. The difficulty here is to define what a "default behaviour" is. A machine will basically do whatever it is designed to do, so an AI-based system too.
  
  We could speculate even more on what would happen if the assistive system was really autonomous, the elusive AGI, but that is really not the case here.
  
  > **Comments:**
  > **username_4**: "The report also said that a human driver would have been unable to make the difference either. "  I haven't read the report, so I could be wrong, but I'm guessing that this wouldn't be true over a *sequence* of frames: there could be subtle cues that an object is present that a human could pick up on, but that image processing would likely dismiss as noise...  
  > **username_3**: Sorry, as I still cannot find the source... The sequence argument is speculative on this case, though. So many things can go wrong with such "early-stage" technology (Who would go to the Moon riding Apollo 11's hardware *today*?). Yes, it may be their algorithm at the time, lack of data, or the hardware is just not enough.  
  
  ---
  
  ### Answer by username_1 on 2016-08-09. Score: 3
  
  As far as I know, Tesla cars autopilot is not a 100% AI pilot, it's an *assitant*: as it detects hands off wheel it slows down, so it's incorrect to speak about AI mistake: it is not trained/designed to drive a car all by itself. A human driver is responsible in that incident.
  
  > **Comments:**
  > **username_2**: How does this answer the actual question?  
  
  ---
  
  ### Answer by username_0 on 2016-08-12. Score: 1
  
  Tesla model S has [Autopilot](https://www.tesla.com/models) which allows to steer within a lane, change lanes with the simple tap of a turn signal, and can manage speed by using traffic-aware cruise control. Multiple digital controls helps to avoid collisions. Based on that, this isn't fully self-driving car.
  
  However it is using a computer vision detection system, but it is not intended to be used hands-free.
  
  So basically what is known is that the accident involved the side of a truck trailer (of a large white 18-wheel truck) and most likely the camera had a washed out of picture possibly due to glare or blooming from overexposure which made that the side of the trailer white and thin which failed to distinguish with the sky which was bright as well.
  
  This may have happened in part, because the crash-avoidance system only engage when both radar and vision system detect an obstacle which could not happen.
  
  Further more it was suggested by *The Associated Press* that the driver most likely was watching a *Harry Potter* at the time of the crash and assuming system would alert Brown, we don't know if he was able to retake controls quickly enough to avoid impact. As mentioned again, the system wasn't intended for hands-free driving and parts of the system was unfinished. Not to mention that the car was driving with full speed under the trailer.
  
  Tesla officially said about this crash in a statement on its website:
  
  > **The high ride height of the trailer combined with its positioning across the road and the extremely rare circumstances** of the impact caused the Model S to pass under the trailer, with the bottom of the trailer impacting the windshield of the Model S.
  >
  > Neither Autopilot nor the driver noticed the white side of the tractor-trailer against a brightly lit sky, so the brake was not applied.
  
  They also said, according to techno-optimists, that they will tweaks their code, so this particular case won't happen again.
  
  To summarize, this was a 'technical failure' of braking system and most likely Autopilot was not at as Tesla told Senate.
  
  [![The New York Times |Source: Florida traffic crash report](https://i.sstatic.net/obdLM.png)](https://i.sstatic.net/obdLM.png)
  
  The New York Times |Source: Florida traffic crash report
  
  Sources:
  
  * [Tesla Autopilot death highlights autonomous risks](http://www.freep.com/story/money/cars/2016/07/01/tesla-autopilot-death-highlights-autonomous-risks/86591130/)
  * [Layers of Autonomy](https://robotfuturesbook.wordpress.com/2016/07/01/layers-of-autonomy)
  * [Inside the Self-Driving Tesla Fatal Accident](http://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html)
  * [Tesla driver dies in first fatal crash while using autopilot mode](https://www.theguardian.com/technology/2016/jun/30/tesla-autopilot-death-self-driving-car-elon-musk)
  
  
  ---
  
  