Answers.Body: <p>Another point of view -<br />
  In safety-critical real world systems, this attack should be evaluated from other aspects as well.<br />
  In many systems the attack is somewhat mitigated to physical attacks only - for example, you can't add digital noise to a camera used for autonomous driving - you need to print an adversarial e.g. stop sign and locate it in a place, where it's still viewed and interpreted incorrectly from several points of view, angles, light and whether conditions, etc.<br />
  Given that, I think that the overall <em>current</em> risk of adversarial examples for scalable attacks on real-world mission-critical systems isn't very high <em>for now</em>.<br />
  That's why such work is existing in companies at research level, but not production, yet.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-03T08:40:00.583Z
Answers.LastActivityDate: 2021-03-03T08:40:00.583Z
Answers.UserDisplayName: rkellerm
Answers.Body: <blockquote>
    <p>However, do industrial strength, production ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?</p>
  </blockquote>
  
  <p>I think it's difficult to tell whether or not there are any <strong>industrial strength</strong> defenses out there (which I assume would mean that they'd be reliable against all or most known methods of attacking). Adversarial Machine Learning is indeed a <a href="https://github.com/yenchenlin/awesome-adversarial-machine-learning" rel="nofollow noreferrer">highly active, and growing, area of research</a>. Not only are new approaches for defending being published quite regularly, but there is also active research into different approaches for "attacking". With new attack methods being discovered frequently, it's unlikely that anyone can already claim to have approaches that would work reliably against them all.</p>
  
  <blockquote>
    <p>The primary interest of this question, however, is whether any <strong>tools</strong> exist that can defend against some adversarial example attacks.</p>
  </blockquote>
  
  <p>The closest thing to a ready-to-use "tool" that I've been able to find is <a href="https://github.com/IBM/adversarial-robustness-toolbox" rel="nofollow noreferrer">IBM's Adversarial Robustness Toolbox</a>, which appears to have various attack and defense methods implemented. It appears to be in active development, which is natural considering the area of research itself is also highly active. I've never tried using it, so I can't vouch personally for the extent to which it's easily usable as a tool for industry, or if it's maybe really only still suitable for research.</p>
  
  <hr>
  
  <p>Based on comments by <a href="https://ai.stackexchange.com/users/16354/ilya-palachev">Ilya</a>, other frameworks that may be useful to consider are <a href="https://github.com/tensorflow/cleverhans" rel="nofollow noreferrer">Cleverhans</a> and <a href="https://github.com/bethgelab/foolbox" rel="nofollow noreferrer">Foolbox</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T16:21:27.227Z
Answers.Text: I'd like to add [CleverHans](https://github.com/tensorflow/cleverhans) and [Foolbox](https://github.com/bethgelab/foolbox), which are the competitors of IBM Adversarial Robustness toolbox. Do you know how these three solutions can be compared?
Answers.UserDisplayName: Ilya Palachev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T17:02:29.790Z
Answers.Text: @IlyaPalachev From a quick glance through their descriptions, I got the impression that those repositories only contain attack methods, not defense methods. I'm not familiar enough with them to tell for sure if that's correct, that's just the impression I got. Obviously, such repositories would still be used for benchmarking defense methods, so searching for places that cite usage of these repositories may lead to interesting defense methods.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T17:43:51.130Z
Answers.Text: CleverHans guys are [implementing](https://github.com/tensorflow/cleverhans/pull/423) defense sub-framework now. In recent [survey](https://dl.acm.org/citation.cfm?id=3134599) about this area they also claim CleverHans to contain "reference  implementations  of several attack and defense procedures" (p. 85).
Answers.UserDisplayName: Ilya Palachev
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T17:56:23.650Z
Answers.Text: @IlyaPalachev Ah I see. For the sake of completeness, I edited them into my answer, where they might be easier to see for future visitors of the site than in the comments.
Answers.UserDisplayName: Dennis Soemers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T16:07:27.653Z
Answers.LastActivityDate: 2018-07-25T17:55:20.797Z
Answers.LastEditDate: 2018-07-25T17:55:20.797Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>The problem of adversarial examples is <a href="https://arxiv.org/pdf/1312.6199.pdf" rel="nofollow noreferrer">known</a> to be critical for neural networks. For example, an image classifier can be manipulated by additively superimposing a different low amplitude image to each of many training examples that looks like noise but is designed to produce specific misclassifications.</p>
  <p><a href="https://i.sstatic.net/5rfe9.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/5rfe9.png" alt="enter image description here" /></a></p>
  <p>Since neural networks are applied to some safety-critical problems (e.g. self-driving cars), I have the following question</p>
  <blockquote>
  <p>What tools are used to ensure safety-critical applications are resistant to the injection of adversarial examples at training time?</p>
  </blockquote>
  <p>Laboratory research aimed at developing defensive security for neural networks exists. These are a few examples.</p>
  <ul>
  <li><p>adversarial training (see e.g. <a href="https://arxiv.org/pdf/1611.01236.pdf" rel="nofollow noreferrer">A. Kurakin et al., ICLR 2017</a>)</p>
  </li>
  <li><p>defensive distillation (see e.g. <a href="https://arxiv.org/pdf/1511.04508.pdf" rel="nofollow noreferrer">N. Papernot et al., SSP 2016</a>)</p>
  </li>
  <li><p>MMSTV defence (<a href="https://arxiv.org/pdf/1706.06083" rel="nofollow noreferrer">Maudry et al., ICLR 2018</a>).</p>
  </li>
  </ul>
  <p>However, do industrial-strength, production-ready defensive strategies and approaches exist?  Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?</p>
  <p>There are already (at least) two questions related to the problem of <a href="https://ai.stackexchange.com/q/6800/2444">hacking</a> and <a href="https://ai.stackexchange.com/q/92/2444">fooling</a> of neural networks. The primary interest of this question, however, is whether any <strong>tools</strong> exist that can defend against some adversarial example attacks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-14T12:04:36.380Z
Comments.Text: [Here](https://ai.stackexchange.com/q/15820/2444) and [here](https://ai.stackexchange.com/q/26448/2444) are two related questions.
Comments.UserDisplayName: nbro
Tags: <resource-request><adversarial-ml><ai-safety><ai-security>
Title: What tools are used to deal with adversarial examples problem?
UserDisplayName: Ilya Palachev
ThreadText: # What tools are used to deal with adversarial examples problem?
  
  (Asked by: username_0 on 2018-06-26)
  
  The problem of adversarial examples is [known](https://arxiv.org/pdf/1312.6199.pdf) to be critical for neural networks. For example, an image classifier can be manipulated by additively superimposing a different low amplitude image to each of many training examples that looks like noise but is designed to produce specific misclassifications.
  
  [![enter image description here](https://i.sstatic.net/5rfe9.png)](https://i.sstatic.net/5rfe9.png)
  
  Since neural networks are applied to some safety-critical problems (e.g. self-driving cars), I have the following question
  
  > What tools are used to ensure safety-critical applications are resistant to the injection of adversarial examples at training time?
  
  Laboratory research aimed at developing defensive security for neural networks exists. These are a few examples.
  
  * adversarial training (see e.g. [A. Kurakin et al., ICLR 2017](https://arxiv.org/pdf/1611.01236.pdf))
  * defensive distillation (see e.g. [N. Papernot et al., SSP 2016](https://arxiv.org/pdf/1511.04508.pdf))
  * MMSTV defence ([Maudry et al., ICLR 2018](https://arxiv.org/pdf/1706.06083)).
  
  However, do industrial-strength, production-ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?
  
  There are already (at least) two questions related to the problem of [hacking](https://ai.stackexchange.com/q/6800/2444) and [fooling](https://ai.stackexchange.com/q/92/2444) of neural networks. The primary interest of this question, however, is whether any **tools** exist that can defend against some adversarial example attacks.
  
  **Tags:** resource-request, adversarial-ml, ai-safety, ai-security
  
  > **Comments:**
  > **username_1**: [Here](https://ai.stackexchange.com/q/15820/2444) and [here](https://ai.stackexchange.com/q/26448/2444) are two related questions.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2018-07-25. Score: 4 | Accepted answer
  
  > However, do industrial strength, production ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?
  
  I think it's difficult to tell whether or not there are any **industrial strength** defenses out there (which I assume would mean that they'd be reliable against all or most known methods of attacking). Adversarial Machine Learning is indeed a [highly active, and growing, area of research](https://github.com/yenchenlin/awesome-adversarial-machine-learning). Not only are new approaches for defending being published quite regularly, but there is also active research into different approaches for "attacking". With new attack methods being discovered frequently, it's unlikely that anyone can already claim to have approaches that would work reliably against them all.
  
  > The primary interest of this question, however, is whether any **tools** exist that can defend against some adversarial example attacks.
  
  The closest thing to a ready-to-use "tool" that I've been able to find is [IBM's Adversarial Robustness Toolbox](https://github.com/IBM/adversarial-robustness-toolbox), which appears to have various attack and defense methods implemented. It appears to be in active development, which is natural considering the area of research itself is also highly active. I've never tried using it, so I can't vouch personally for the extent to which it's easily usable as a tool for industry, or if it's maybe really only still suitable for research.
  
  ---
  
  Based on comments by [username_0](https://ai.stackexchange.com/users/16354/username_0), other frameworks that may be useful to consider are [Cleverhans](https://github.com/tensorflow/cleverhans) and [Foolbox](https://github.com/bethgelab/foolbox).
  
  > **Comments:**
  > **username_0**: I'd like to add [CleverHans](https://github.com/tensorflow/cleverhans) and [Foolbox](https://github.com/bethgelab/foolbox), which are the competitors of IBM Adversarial Robustness toolbox. Do you know how these three solutions can be compared?  
  > **username_3**: @username_0 From a quick glance through their descriptions, I got the impression that those repositories only contain attack methods, not defense methods. I'm not familiar enough with them to tell for sure if that's correct, that's just the impression I got. Obviously, such repositories would still be used for benchmarking defense methods, so searching for places that cite usage of these repositories may lead to interesting defense methods.  
  > **username_0**: CleverHans guys are [implementing](https://github.com/tensorflow/cleverhans/pull/423) defense sub-framework now. In recent [survey](https://dl.acm.org/citation.cfm?id=3134599) about this area they also claim CleverHans to contain "reference  implementations  of several attack and defense procedures" (p. 85).  
  > **username_3**: @username_0 Ah I see. For the sake of completeness, I edited them into my answer, where they might be easier to see for future visitors of the site than in the comments.  
  
  ---
  
  ### Answer by username_2 on 2021-03-03. Score: 1
  
  Another point of view -  
  In safety-critical real world systems, this attack should be evaluated from other aspects as well.  
  In many systems the attack is somewhat mitigated to physical attacks only - for example, you can't add digital noise to a camera used for autonomous driving - you need to print an adversarial e.g. stop sign and locate it in a place, where it's still viewed and interpreted incorrectly from several points of view, angles, light and whether conditions, etc.  
  Given that, I think that the overall *current* risk of adversarial examples for scalable attacks on real-world mission-critical systems isn't very high *for now*.  
  That's why such work is existing in companies at research level, but not production, yet.
  
  
  ---
  
  