Answers.Body: <p>AI is vulnerable from two security perspectives the way I see it: </p>
  
  <ol>
  <li><p>The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data.</p></li>
  <li><p>Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with.</p></li>
  </ol>
  
  <p>The first has to be mitigated in the same way as any other software. I'm uncertain if AI is any more vulnerable on this front than other software, I'd be inclined to think that the complexity maybe slightly heightens the risk.</p>
  
  <p>The second is probably best mitigated by both the careful refinement of the system as noted in some of the other answers, but also by making the system more context-sensitive; many adversarial techniques rely on the input being assessed in a vacuum. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-20T09:10:59.067Z
Answers.Text: The split between code vulnerabilities and usage vulnerabilities is good. However, the code vulnerabilities typically are minuscule in AI. The complexity of AI lies in the data, whether that's node weights in a neural network or trees in a random forest. There's just a small bit of code to feed the AI, and the chief risk there is not overfeeding it - a classic buffer overflow risk, easily mitigated by late 20th century techniques.
Answers.UserDisplayName: MSalters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T16:47:44.133Z
Answers.Text: @MSalters I think it is hard to draw a general conclusion because the code complexity can vary a lot between different types of AI agents (I think your comment is largely accurate for neural networks). Furthermore, although the data and manipulation thereof is probably the larger attack surface it would be unwise to discount the same sort of attacks that have allowed remote code execution via compromised image files in the past that exploited flaws in image viewing applications. The vector is the data being passed in, but the behavior still falls under the code vulnerability header, I think.
Answers.UserDisplayName: Christopher Griffith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T17:13:58.507Z
Answers.LastActivityDate: 2018-06-19T17:13:58.507Z
Answers.UserDisplayName: Christopher Griffith
Answers.Body: <h2>Programmer vs Programmer</h2>
  
  <p>It's a "infinity war": Programmers vs Programmers. All thing can be hackable. Prevention is linked to the level of knowledge of the professional in charge of security and programmers in application security.</p>
  
  <p><strong>eg</strong> There are several ways to identify a user trying to mess up the metrics generated by Sentiment Analysis, but there are ways to circumvent those steps as well. It's a pretty boring fight.</p>
  
  <h2>Agent vs Agent</h2>
  
  <p>An interesting point that @DukeZhou raised is the evolution of this war, involving two artificial intelligence (agents). In that case, the battle is one of the most knowledgeable. Which is the best-trained model, you know?</p>
  
  <p>However, to achieve perfection in the issue of vulnerability, artificial intelligence or artificial super intelligence surpass the ability to circumvent the human. It is as if the knowledge of all hacks to this day already existed in the mind of this agent and he began to develop new ways of circumventing his own system and developing protection. Complex, right?</p>
  
  <p>I believe it's hard to have an AI who thinks: "Will the human going to use a photo instead of putting his face to be identified?"</p>
  
  <h2>How we can prevent it</h2>
  
  <p>Always having a human supervising the machine, and yet it will not be 100% effective. This disregarding the possibility that an agent can improve his own model alone.</p>
  
  <h2>Conclusion</h2>
  
  <p>So I think the scenario works this way: a programmer tries to circumvent the validations of an AI and the IA developer acquiring knowledge through logs and tests tries to build a smarter and safer model trying to reduce the chances of failure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T16:35:51.970Z
Answers.LastActivityDate: 2018-07-03T14:49:14.257Z
Answers.LastEditDate: 2018-07-03T14:49:14.257Z
Answers.UserDisplayName: GIA
Answers.Body: <blockquote>
    <p>How we can prevent it?</p>
  </blockquote>
  
  <p>There are several works about AI verification. Automatic verifiers can prove the robustness properties of neural networks. It means that if the input X of the NN is perturbed not more that on a given limit ε (in some metric, e.g. L2), then the NN gives the same answer on it.</p>
  
  <p>Such verifiers are done by:</p>
  
  <ul>
  <li>Stanford: <a href="https://arxiv.org/pdf/1702.01135.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1702.01135.pdf</a></li>
  <li>ETHZ: <a href="https://www.sri.inf.ethz.ch/papers/sp2018.pdf" rel="nofollow noreferrer">https://www.sri.inf.ethz.ch/papers/sp2018.pdf</a></li>
  <li>Google: <a href="https://arxiv.org/pdf/1803.06567.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1803.06567.pdf</a>, <a href="https://arxiv.org/pdf/1805.10265.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1805.10265.pdf</a></li>
  <li>Bosch: <a href="https://arxiv.org/pdf/1805.10265.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1805.10265.pdf</a></li>
  </ul>
  
  <p>This approach may help to check robustness properties of neural networks. The next step is to construct such a neural network, that has required robustness. Some of above papers contain also methods of how to do that.</p>
  
  <p>There are different techniques to improve the robustness of neural networks:</p>
  
  <ul>
  <li><p>adversarial training (see e.g. <a href="https://arxiv.org/pdf/1611.01236.pdf" rel="nofollow noreferrer">A. Kurakin et al., ICLR 2017</a>)</p></li>
  <li><p>defensive distillation (see e.g. <a href="https://arxiv.org/pdf/1511.04508.pdf" rel="nofollow noreferrer">N. Papernot et al., SSP 2016</a>)</p></li>
  <li><p>MMSTV defence (<a href="https://arxiv.org/pdf/1706.06083" rel="nofollow noreferrer">Maudry et al., ICLR 2018</a>).</p></li>
  </ul>
  
  <p>At least the last one can provably make NN more robust. More literature can be found <a href="https://github.com/TrustAI/Literature-on-DNN-Verification-and-Testing/blob/master/defences.md" rel="nofollow noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T21:54:00.973Z
Answers.Text: This sounds like an impossible claim... unless it's about some particular inputs X, rather than general inputs X? In which case, it seems to say next to nothing about hackability, since inputs need not be limited to perturbations of those in the training?
Answers.UserDisplayName: user541686
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-20T09:15:09.513Z
Answers.Text: @Mehrdad: It's probably achievable in a probabilistic sense if the input space is sufficiently structured that you can randomly sample it. That is to say, you can probably establish that for 95% of possible inputs, 95% of disturbances smaller than ε do not affect the class label. This is equivalent to establishing that the border between output classes in input space is smooth, or that the largest part of the input space does not lie near a class border. Obviously some part of the input space has to lie near a class border.
Answers.UserDisplayName: MSalters
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-20T10:35:44.453Z
Answers.Text: I'm not sure this would apply in the "adversarial" case described in the paper: There, (IIRC) a back-propagated gradient is added to the whole picture, so the change to the complete input can be quite large - even if the change for each *individual* pixel is barely noticeable.
Answers.UserDisplayName: Niki
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-20T16:17:36.013Z
Answers.Text: @MSalters: I guess, yeah. But then that seems to devalue it a fair bit unless you can actually show the pictures that are on a class border should actually *be* on a class border...
Answers.UserDisplayName: user541686
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-21T17:29:16.020Z
Answers.Text: The sentence "The next step is to construct such a neural network, that has required robustness" is under research. In general it's very hard to get rid of NN non-robustness problem. But it's possible to enhance the robustness by adversarial training (see e.g. [A. Kurakin et al., ICLR 2017](https://arxiv.org/pdf/1611.01236.pdf)), defensive distillation (see e.g. [N. Papernot et al., SSP 2016](https://arxiv.org/pdf/1511.04508.pdf)), MMSTV defence ([Maudry et al., ICLR 2018](https://arxiv.org/pdf/1706.06083)). At least the last one can provably make NN more robust.
Answers.UserDisplayName: Ilya Palachev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T09:18:06.403Z
Answers.Text: @Mehrard Yes, inputs need to be limited to those in the training. If we don't provide robustness on the train set, it is meaningless to say anything about robustness in the general case. So those works start from the trivial goal: provide robustness at least for the train/test set. Even this trivial goal is not trivial to reach.
Answers.UserDisplayName: Ilya Palachev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T14:48:33.327Z
Answers.LastActivityDate: 2018-06-21T17:31:16.403Z
Answers.LastEditDate: 2018-06-21T17:31:16.403Z
Answers.UserDisplayName: Ilya Palachev
Answers.Body: <p>I believe it is, no system is safe, however I am not sure if I can still say this after 20-30 years of AI development/evolution. Anyways, there are articles that showed humans fooling AI (Computer Vision).</p>
  
  <p><a href="https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google" rel="nofollow noreferrer">https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google</a></p>
  
  <p><a href="https://spectrum.ieee.org/cars-that-think/transportation/sensors/slight-street-sign-modifications-can-fool-machine-learning-algorithms" rel="nofollow noreferrer">https://spectrum.ieee.org/cars-that-think/transportation/sensors/slight-street-sign-modifications-can-fool-machine-learning-algorithms</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T13:41:18.600Z
Answers.LastActivityDate: 2018-06-19T13:41:18.600Z
Answers.UserDisplayName: Akio
Answers.Body: <blockquote>
    <p>Is Artificial Intelligence Vulnerable to Hacking?</p>
  </blockquote>
  
  <p>Invert your question for a moment and think:</p>
  
  <blockquote>
    <p>What would make AI at less of a risk of hacking compared to any other
    kind of software?</p>
  </blockquote>
  
  <p>At the end of the day, software is software and there will always be bugs and security issues. AIs are at risk to all the problems non-AI software is at risk to, being AI doesn't grant it some kind of immunity.</p>
  
  <p>As for AI-specific tampering, AI is at risk to being fed false information. Unlike most programs, AI's functionality is determined by the data it consumes.</p>
  
  <p>For a real world example, a few years ago Microsoft created an AI chatbot called Tay. It took the people of Twitter less than 24 hours to teach it to say "We're going to build a wall, and mexico is going to pay for it":</p>
  
  <p><a href="https://i.sstatic.net/ZvUkp.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZvUkp.jpg" alt="We&#39;re going to build a wall, and mexico is going to pay for it"></a></p>
  
  <p>(Image taken from the Verge article linked below, I claim no credit for it.)</p>
  
  <p>And that's just the tip of the iceberg.</p>
  
  <p>Some articles about Tay:</p>
  
  <ul>
  <li><a href="https://www.bbc.co.uk/news/technology-35890188" rel="nofollow noreferrer">BBC</a></li>
  <li><a href="https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist" rel="nofollow noreferrer">The Verge</a></li>
  </ul>
  
  <p>Now imagine that wasn't a chat bot, imagine that was an important piece of AI from a future where AI are in charge of things like not killing the occupants of a car (i.e. a self-driving car) or not killing a patient on the operating table (i.e. some kind of medical assistance equipment).</p>
  
  <p>Granted, one would hope such AIs would be better secured against such threats, but supposing someone did find a way to feed such an AI masses of false information without being noticed (after all, the best hackers leave no trace), that genuinely could mean the difference between life and death.</p>
  
  <p>Using the example of a self-driving car, imagine if false data could make the car think it needed to do an emergency stop when on a motorway. One of the applications for medical AI is life-or-death decisions in the ER, imagine if a hacker could tip the scales in favour of the wrong decision.</p>
  
  <blockquote>
    <p>How we can prevent it?</p>
  </blockquote>
  
  <p>Ultimately the scale of the risk depends on how reliant humans become on AI. For example, if humans took the judgement of an AI and never questioned it, they'd be opening themselves up to all sorts of manipulation. However, if they use the AI's analysis as just one part of the puzzle, it would become easier to spot when an AI is wrong, be it through accidental or malicious means.</p>
  
  <p>In the case of a medical decision maker, don't just believe the AI, carry out physical tests and get some human opinions too. If two doctors disagree with the AI, throw out the AI's diagnosis.</p>
  
  <p>In the case of a car, one possibility is to have several redundant systems that must essentially 'vote' about what to do. If a car had multiple AIs on separate systems that must vote about which action to take, a hacker would have to take out more than just one AI to get control or cause a stalemate. Importantly, if the AIs ran on different systems, the same exploitation used on one couldn't be done on another, further increasing the hacker's workload.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T17:37:40.237Z
Answers.Text: I like the idea of having several separate AI systems that have to reach agreement as a mitigation technique. Although then you'd have to be assured whatever voting mechanism they used couldn't be comprised to fake a decision.
Answers.UserDisplayName: Christopher Griffith
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-22T22:26:57.043Z
Answers.Text: @ChristopherGriffith True, that is a risk. In the case of the car, the best way to mitigate that is to design the system so that an attacker would need physical access to manipulate it and make it hard to reach so the person would have to break in to the car to access it. Keeping a system offline is generally a good hacking countermeasure, though not always ideal.
Answers.UserDisplayName: Pharap
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-20T01:31:16.650Z
Answers.LastActivityDate: 2018-06-20T01:31:16.650Z
Answers.UserDisplayName: Pharap
Answers.Body: <p>I concur with Akio that no system is completely safe, but the take away is AI systems are less prone to attacks when comparing with the old systems because of the ability to constantly improve.</p>
  
  <p>As time passes by more people will get in the field bringing new ideas and hardware will be improving so that they are "strong AI."</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T14:10:41.863Z
Answers.LastActivityDate: 2018-06-19T18:41:06.783Z
Answers.LastEditDate: 2018-06-19T18:41:06.783Z
Answers.UserDisplayName: Simbarashe Timothy Motsi
Answers.Body: <p>There are many ways to hack an AI. When I was kid I figured how to beat a chess computer. I always followed the same pattern, once you learn you can exploit it. The worlds best hacker is a 4 year old that wants something he will try different things until he establishes pattern in his parents. Anyway, Get an Ai to learn the patterns of a AI and given a given combination you can figure the outcome. There is also just plain flaws or back door in code either on purpose or by chance. There is also the possibility the AI will hack itself. It is called misbehaving, remember the small child again...</p>
  
  <p>BTW simple way is to make AI always fails safe... something people forget.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-19T23:57:07.290Z
Answers.LastActivityDate: 2018-06-19T23:57:07.290Z
Answers.UserDisplayName: Tony Lester
Body: <p>The paper <a href="https://arxiv.org/pdf/1511.07528.pdf" rel="nofollow noreferrer"><em>The Limitations of Deep Learning in Adversarial Settings</em></a> explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. The authors experiment with a neural network meant to read handwritten digits, undermining its reading ability by distorting the samples of handwritten digits that the neural network is trained with.</p>
  
  <p>I'm concerned that malicious actors might try hacking AI. For example</p>
  
  <ul>
  <li>Fooling autonomous vehicles to misinterpret stop signs vs. speed limit.</li>
  <li>Bypassing facial recognition, such as the ones for ATM.</li>
  <li>Bypassing spam filters.</li>
  <li>Fooling sentiment analysis of movie reviews, hotels, etc.</li>
  <li>Bypassing anomaly detection engines.</li>
  <li>Faking voice commands.</li>
  <li>Misclassifying machine learning based-medical predictions.</li>
  </ul>
  
  <p>What adversarial effect could disrupt the world? How we can prevent it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-20T12:10:29.887Z
Comments.Text: Consider that _human_ intelligence is vulnerable to hacking
Comments.UserDisplayName: Gaius
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-13T01:12:12.963Z
Comments.Text: Interesting. Are you interested in "adversarial settings risk models" or something closer to a traditional cyber-security answer but still squarely about A.I.? Best wishes.
Comments.UserDisplayName: Donate to the Edhi Foundation
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-18T19:58:57.367Z
Comments.Text: It's not clear what you're really asking here, to be honest. You're asking too many questions or none, and that's why I've read below many different answers. This type of post is not good for Stack Exchange. We prefer more specific questions. What is your **main specific** question here? You're listing many possible ways to attack AI systems and then you ask "Are AI systems vulnerable to hacking?". This doesn't make any sense. What makes even less sense, because not really the same question, is the question "What adversarial effect could disrupt the world?".
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-14T12:59:28.653Z
Comments.Text: This post is "interesting", but I closed it because it's unclear what type of answer you're looking for and it seems that you're asking many questions. There seem to be 3 distinct questions here. 1. The one in the title "_Is artificial intelligence vulnerable to hacking?_", 2. "_What adversarial effect could disrupt the world?_" and 3. "_How we can prevent it?_". It seems that you're proposing ways in which AI systems could be hacked, so your question in the title is unclear. Are you looking for a "yes" or "no" answer?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-14T13:03:59.933Z
Comments.Text: Do you want us to confirm that AI systems can be hacked in the way you suggest? Again, yes or no answer? The 2nd question is even more problematic. What do you mean by "disrupt the world" and "adversarial effect"? Maybe you meant "adversarial attack" rather than "adversarial effect"? Even assuming that's the case, it's not clear what you mean by "disrupt the world". It could mean many things. The 3rd question may be ok, if you clarify the second question. I would suggest that you edit your post to clarify what your question really is.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><ai-security><ai-safety><adversarial-ml>
Title: Is artificial intelligence vulnerable to hacking?
UserDisplayName: Surya Sg
ThreadText: # Is artificial intelligence vulnerable to hacking?
  
  (Asked by: username_0 on 2018-06-19)
  
  The paper [*The Limitations of Deep Learning in Adversarial Settings*](https://arxiv.org/pdf/1511.07528.pdf) explores how neural networks might be corrupted by an attacker who can manipulate the data set that the neural network trains with. The authors experiment with a neural network meant to read handwritten digits, undermining its reading ability by distorting the samples of handwritten digits that the neural network is trained with.
  
  I'm concerned that malicious actors might try hacking AI. For example
  
  * Fooling autonomous vehicles to misinterpret stop signs vs. speed limit.
  * Bypassing facial recognition, such as the ones for ATM.
  * Bypassing spam filters.
  * Fooling sentiment analysis of movie reviews, hotels, etc.
  * Bypassing anomaly detection engines.
  * Faking voice commands.
  * Misclassifying machine learning based-medical predictions.
  
  What adversarial effect could disrupt the world? How we can prevent it?
  
  **Tags:** neural-networks, machine-learning, ai-security, ai-safety, adversarial-ml
  > **Comments:**
  > **username_1**: Consider that _human_ intelligence is vulnerable to hacking  
  > **username_2**: Interesting. Are you interested in "adversarial settings risk models" or something closer to a traditional cyber-security answer but still squarely about A.I.? Best wishes.  
  > **username_3**: It's not clear what you're really asking here, to be honest. You're asking too many questions or none, and that's why I've read below many different answers. This type of post is not good for Stack Exchange. We prefer more specific questions. What is your **main specific** question here? You're listing many possible ways to attack AI systems and then you ask "Are AI systems vulnerable to hacking?". This doesn't make any sense. What makes even less sense, because not really the same question, is the question "What adversarial effect could disrupt the world?".  
  > **username_3**: This post is "interesting", but I closed it because it's unclear what type of answer you're looking for and it seems that you're asking many questions. There seem to be 3 distinct questions here. 1. The one in the title "_Is artificial intelligence vulnerable to hacking?_", 2. "_What adversarial effect could disrupt the world?_" and 3. "_How we can prevent it?_". It seems that you're proposing ways in which AI systems could be hacked, so your question in the title is unclear. Are you looking for a "yes" or "no" answer?  
  > **username_3**: Do you want us to confirm that AI systems can be hacked in the way you suggest? Again, yes or no answer? The 2nd question is even more problematic. What do you mean by "disrupt the world" and "adversarial effect"? Maybe you meant "adversarial attack" rather than "adversarial effect"? Even assuming that's the case, it's not clear what you mean by "disrupt the world". It could mean many things. The 3rd question may be ok, if you clarify the second question. I would suggest that you edit your post to clarify what your question really is.  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_4 on 2018-06-19. Score: 21
  
  AI is vulnerable from two security perspectives the way I see it:
  
  1. The classic method of exploiting outright programmatic errors to achieve some sort of code execution on the machine that is running the AI or to extract data.
  2. Trickery through the equivalent of AI optical illusions for the particular form of data that the system is designed to deal with.
  
  The first has to be mitigated in the same way as any other software. I'm uncertain if AI is any more vulnerable on this front than other software, I'd be inclined to think that the complexity maybe slightly heightens the risk.
  
  The second is probably best mitigated by both the careful refinement of the system as noted in some of the other answers, but also by making the system more context-sensitive; many adversarial techniques rely on the input being assessed in a vacuum.
  
  > **Comments:**
  > **username_5**: The split between code vulnerabilities and usage vulnerabilities is good. However, the code vulnerabilities typically are minuscule in AI. The complexity of AI lies in the data, whether that's node weights in a neural network or trees in a random forest. There's just a small bit of code to feed the AI, and the chief risk there is not overfeeding it - a classic buffer overflow risk, easily mitigated by late 20th century techniques.  
  > **username_4**: @MSalters I think it is hard to draw a general conclusion because the code complexity can vary a lot between different types of AI agents (I think your comment is largely accurate for neural networks). Furthermore, although the data and manipulation thereof is probably the larger attack surface it would be unwise to discount the same sort of attacks that have allowed remote code execution via compromised image files in the past that exploited flaws in image viewing applications. The vector is the data being passed in, but the behavior still falls under the code vulnerability header, I think.  
  
  ---
  
  ### Answer by username_6 on 2018-06-19. Score: 7
  
  ## Programmer vs Programmer
  
  It's a "infinity war": Programmers vs Programmers. All thing can be hackable. Prevention is linked to the level of knowledge of the professional in charge of security and programmers in application security.
  
  **eg** There are several ways to identify a user trying to mess up the metrics generated by Sentiment Analysis, but there are ways to circumvent those steps as well. It's a pretty boring fight.
  
  ## Agent vs Agent
  
  An interesting point that @DukeZhou raised is the evolution of this war, involving two artificial intelligence (agents). In that case, the battle is one of the most knowledgeable. Which is the best-trained model, you know?
  
  However, to achieve perfection in the issue of vulnerability, artificial intelligence or artificial super intelligence surpass the ability to circumvent the human. It is as if the knowledge of all hacks to this day already existed in the mind of this agent and he began to develop new ways of circumventing his own system and developing protection. Complex, right?
  
  I believe it's hard to have an AI who thinks: "Will the human going to use a photo instead of putting his face to be identified?"
  
  ## How we can prevent it
  
  Always having a human supervising the machine, and yet it will not be 100% effective. This disregarding the possibility that an agent can improve his own model alone.
  
  ## Conclusion
  
  So I think the scenario works this way: a programmer tries to circumvent the validations of an AI and the IA developer acquiring knowledge through logs and tests tries to build a smarter and safer model trying to reduce the chances of failure.
  
  
  ---
  
  ### Answer by username_7 on 2018-06-19. Score: 6
  
  > How we can prevent it?
  
  There are several works about AI verification. Automatic verifiers can prove the robustness properties of neural networks. It means that if the input X of the NN is perturbed not more that on a given limit ε (in some metric, e.g. L2), then the NN gives the same answer on it.
  
  Such verifiers are done by:
  
  * Stanford: <https://arxiv.org/pdf/1702.01135.pdf>
  * ETHZ: <https://www.sri.inf.ethz.ch/papers/sp2018.pdf>
  * Google: <https://arxiv.org/pdf/1803.06567.pdf>, <https://arxiv.org/pdf/1805.10265.pdf>
  * Bosch: <https://arxiv.org/pdf/1805.10265.pdf>
  
  This approach may help to check robustness properties of neural networks. The next step is to construct such a neural network, that has required robustness. Some of above papers contain also methods of how to do that.
  
  There are different techniques to improve the robustness of neural networks:
  
  * adversarial training (see e.g. [A. Kurakin et al., ICLR 2017](https://arxiv.org/pdf/1611.01236.pdf))
  * defensive distillation (see e.g. [N. Papernot et al., SSP 2016](https://arxiv.org/pdf/1511.04508.pdf))
  * MMSTV defence ([Maudry et al., ICLR 2018](https://arxiv.org/pdf/1706.06083)).
  
  At least the last one can provably make NN more robust. More literature can be found [here](https://github.com/TrustAI/Literature-on-DNN-Verification-and-Testing/blob/master/defences.md).
  
  > **Comments:**
  > **username_8**: This sounds like an impossible claim... unless it's about some particular inputs X, rather than general inputs X? In which case, it seems to say next to nothing about hackability, since inputs need not be limited to perturbations of those in the training?  
  > **username_5**: @Mehrdad: It's probably achievable in a probabilistic sense if the input space is sufficiently structured that you can randomly sample it. That is to say, you can probably establish that for 95% of possible inputs, 95% of disturbances smaller than ε do not affect the class label. This is equivalent to establishing that the border between output classes in input space is smooth, or that the largest part of the input space does not lie near a class border. Obviously some part of the input space has to lie near a class border.  
  > **username_9**: I'm not sure this would apply in the "adversarial" case described in the paper: There, (IIRC) a back-propagated gradient is added to the whole picture, so the change to the complete input can be quite large - even if the change for each *individual* pixel is barely noticeable.  
  > **username_8**: @MSalters: I guess, yeah. But then that seems to devalue it a fair bit unless you can actually show the pictures that are on a class border should actually *be* on a class border...  
  > **username_7**: The sentence "The next step is to construct such a neural network, that has required robustness" is under research. In general it's very hard to get rid of NN non-robustness problem. But it's possible to enhance the robustness by adversarial training (see e.g. [A. Kurakin et al., ICLR 2017](https://arxiv.org/pdf/1611.01236.pdf)), defensive distillation (see e.g. [N. Papernot et al., SSP 2016](https://arxiv.org/pdf/1511.04508.pdf)), MMSTV defence ([Maudry et al., ICLR 2018](https://arxiv.org/pdf/1706.06083)). At least the last one can provably make NN more robust.  
  > **username_7**: @Mehrard Yes, inputs need to be limited to those in the training. If we don't provide robustness on the train set, it is meaningless to say anything about robustness in the general case. So those works start from the trivial goal: provide robustness at least for the train/test set. Even this trivial goal is not trivial to reach.  
  
  ---
  
  ### Answer by username_10 on 2018-06-19. Score: 4
  
  I believe it is, no system is safe, however I am not sure if I can still say this after 20-30 years of AI development/evolution. Anyways, there are articles that showed humans fooling AI (Computer Vision).
  
  <https://www.theverge.com/2018/1/3/16844842/ai-computer-vision-trick-adversarial-patches-google>
  
  <https://spectrum.ieee.org/cars-that-think/transportation/sensors/slight-street-sign-modifications-can-fool-machine-learning-algorithms>
  
  
  ---
  
  ### Answer by username_11 on 2018-06-20. Score: 4
  
  > Is Artificial Intelligence Vulnerable to Hacking?
  
  Invert your question for a moment and think:
  
  > What would make AI at less of a risk of hacking compared to any other
  > kind of software?
  
  At the end of the day, software is software and there will always be bugs and security issues. AIs are at risk to all the problems non-AI software is at risk to, being AI doesn't grant it some kind of immunity.
  
  As for AI-specific tampering, AI is at risk to being fed false information. Unlike most programs, AI's functionality is determined by the data it consumes.
  
  For a real world example, a few years ago Microsoft created an AI chatbot called Tay. It took the people of Twitter less than 24 hours to teach it to say "We're going to build a wall, and mexico is going to pay for it":
  
  [![We're going to build a wall, and mexico is going to pay for it](https://i.sstatic.net/ZvUkp.jpg)](https://i.sstatic.net/ZvUkp.jpg)
  
  (Image taken from the Verge article linked below, I claim no credit for it.)
  
  And that's just the tip of the iceberg.
  
  Some articles about Tay:
  
  * [BBC](https://www.bbc.co.uk/news/technology-35890188)
  * [The Verge](https://www.theverge.com/2016/3/24/11297050/tay-microsoft-chatbot-racist)
  
  Now imagine that wasn't a chat bot, imagine that was an important piece of AI from a future where AI are in charge of things like not killing the occupants of a car (i.e. a self-driving car) or not killing a patient on the operating table (i.e. some kind of medical assistance equipment).
  
  Granted, one would hope such AIs would be better secured against such threats, but supposing someone did find a way to feed such an AI masses of false information without being noticed (after all, the best hackers leave no trace), that genuinely could mean the difference between life and death.
  
  Using the example of a self-driving car, imagine if false data could make the car think it needed to do an emergency stop when on a motorway. One of the applications for medical AI is life-or-death decisions in the ER, imagine if a hacker could tip the scales in favour of the wrong decision.
  
  > How we can prevent it?
  
  Ultimately the scale of the risk depends on how reliant humans become on AI. For example, if humans took the judgement of an AI and never questioned it, they'd be opening themselves up to all sorts of manipulation. However, if they use the AI's analysis as just one part of the puzzle, it would become easier to spot when an AI is wrong, be it through accidental or malicious means.
  
  In the case of a medical decision maker, don't just believe the AI, carry out physical tests and get some human opinions too. If two doctors disagree with the AI, throw out the AI's diagnosis.
  
  In the case of a car, one possibility is to have several redundant systems that must essentially 'vote' about what to do. If a car had multiple AIs on separate systems that must vote about which action to take, a hacker would have to take out more than just one AI to get control or cause a stalemate. Importantly, if the AIs ran on different systems, the same exploitation used on one couldn't be done on another, further increasing the hacker's workload.
  
  > **Comments:**
  > **username_4**: I like the idea of having several separate AI systems that have to reach agreement as a mitigation technique. Although then you'd have to be assured whatever voting mechanism they used couldn't be comprised to fake a decision.  
  > **username_11**: @ChristopherGriffith True, that is a risk. In the case of the car, the best way to mitigate that is to design the system so that an attacker would need physical access to manipulate it and make it hard to reach so the person would have to break in to the car to access it. Keeping a system offline is generally a good hacking countermeasure, though not always ideal.  
  
  ---
  
  ### Answer by username_12 on 2018-06-19. Score: 1
  
  I concur with Akio that no system is completely safe, but the take away is AI systems are less prone to attacks when comparing with the old systems because of the ability to constantly improve.
  
  As time passes by more people will get in the field bringing new ideas and hardware will be improving so that they are "strong AI."
  
  
  ---
  
  ### Answer by username_13 on 2018-06-19. Score: 0
  
  There are many ways to hack an AI. When I was kid I figured how to beat a chess computer. I always followed the same pattern, once you learn you can exploit it. The worlds best hacker is a 4 year old that wants something he will try different things until he establishes pattern in his parents. Anyway, Get an Ai to learn the patterns of a AI and given a given combination you can figure the outcome. There is also just plain flaws or back door in code either on purpose or by chance. There is also the possibility the AI will hack itself. It is called misbehaving, remember the small child again...
  
  BTW simple way is to make AI always fails safe... something people forget.
  
  
  ---
  
  