Answers.Body: <p>These are known as <strong>adversarial attacks</strong>, and the specific examples that are misclassified are known as <strong>adversarial examples</strong>.</p>
  <p>There is a reasonably large body of work on finding adversarial examples, and on making CNNs more robust (i.e. less prone to these attacks). An example is the <a href="https://arxiv.org/pdf/1511.04599.pdf" rel="noreferrer">DeepFool</a> algorithm, which can be used to find perturbations of data which would cause the label to change.</p>
  <p>There are several techniques in the literature which are used to fight against adversarial attacks. Here is a selection:</p>
  <ol>
  <li><p>Augmenting the training data with various random perturbations. This is intended to make the model more robust to the typical adversarial attack where random noise is added to an image. An example of this approach is discussed in [1].</p>
  </li>
  <li><p>Constructing some sort of model to &quot;denoise&quot; input before feeding into the CNN. An example of this is Defense-GAN [2], which uses a generative adversarial model which models the &quot;true&quot; image distribution and finds an approximation of the input closer to the real distribution.</p>
  </li>
  </ol>
  <h3>References</h3>
  <p>[1] Ian J. Goodfellow, Jonathon Shlens &amp; Christian Szegedy. <em>Explaining and Harnessing Adversarial Examples</em>. ICLR (2015). <a href="https://arxiv.org/pdf/1412.6572.pdf" rel="noreferrer">URL</a>.</p>
  <p>[2] Pouya Samangouei, Maya Kabkab, Rama Chellappa. <em>Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models</em>. ICLR (2018). <a href="https://arxiv.org/pdf/1805.06605.pdf" rel="noreferrer">URL</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-18T19:28:37.933Z
Answers.LastActivityDate: 2021-02-18T19:28:37.933Z
Answers.UserDisplayName: htl
Body: <p>I while ago I read that you can make subtle changes to an image that will ensure a good CNN will horribly misclassify the image. I believe the changes must exploit details of the CNN that will be used for classification. So we can trick a good CNN into classifying an image as a picture of a bicycle when any human would say it's an image of a dog. What do we call that technique, and is there an effort to make image classifiers robust against this trick?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-18T19:43:42.533Z
Comments.Text: This seems to be at least a partial duplicate of this: https://ai.stackexchange.com/q/15820/2444 (how can we generate inputs to an AI to fool them? is there any research on this?) or this https://ai.stackexchange.com/q/6800/2444 (in what ways could you fool an "AI", so not just a neural network, so this question seems to be too broad), or this https://ai.stackexchange.com/q/92/2444 (how is that possible that you can fool a NN), or this https://ai.stackexchange.com/q/6892/2444 (what tools are used to deal with AML).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-18T20:07:29.333Z
Comments.Text: However, currently, none of these seem to be exact duplicates, so I will leave this open for a while, because your question focuses on **CNNs**, and I hope that the answers below understand that.
Comments.UserDisplayName: nbro
Tags: <convolutional-neural-networks><reference-request><terminology><adversarial-ml>
Title: Can CNNs be made robust to tricks where small changes cause misclassification?
UserDisplayName: Ted Ersek
ThreadText: # Can CNNs be made robust to tricks where small changes cause misclassification?
  
  (Asked by: username_0 on 2021-02-18)
  
  I while ago I read that you can make subtle changes to an image that will ensure a good CNN will horribly misclassify the image. I believe the changes must exploit details of the CNN that will be used for classification. So we can trick a good CNN into classifying an image as a picture of a bicycle when any human would say it's an image of a dog. What do we call that technique, and is there an effort to make image classifiers robust against this trick?
  
  **Tags:** convolutional-neural-networks, reference-request, terminology, adversarial-ml
  
  > **Comments:**
  > **username_1**: This seems to be at least a partial duplicate of this: https://ai.stackexchange.com/q/15820/2444 (how can we generate inputs to an AI to fool them? is there any research on this?) or this https://ai.stackexchange.com/q/6800/2444 (in what ways could you fool an "AI", so not just a neural network, so this question seems to be too broad), or this https://ai.stackexchange.com/q/92/2444 (how is that possible that you can fool a NN), or this https://ai.stackexchange.com/q/6892/2444 (what tools are used to deal with AML).  
  > **username_1**: However, currently, none of these seem to be exact duplicates, so I will leave this open for a while, because your question focuses on **CNNs**, and I hope that the answers below understand that.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-02-18. Score: 5 | Accepted answer
  
  These are known as **adversarial attacks**, and the specific examples that are misclassified are known as **adversarial examples**.
  
  There is a reasonably large body of work on finding adversarial examples, and on making CNNs more robust (i.e. less prone to these attacks). An example is the [DeepFool](https://arxiv.org/pdf/1511.04599.pdf) algorithm, which can be used to find perturbations of data which would cause the label to change.
  
  There are several techniques in the literature which are used to fight against adversarial attacks. Here is a selection:
  
  1. Augmenting the training data with various random perturbations. This is intended to make the model more robust to the typical adversarial attack where random noise is added to an image. An example of this approach is discussed in [1].
  2. Constructing some sort of model to "denoise" input before feeding into the CNN. An example of this is Defense-GAN [2], which uses a generative adversarial model which models the "true" image distribution and finds an approximation of the input closer to the real distribution.
  
  ### References
  
  [1] Ian J. Goodfellow, Jonathon Shlens & Christian Szegedy. *Explaining and Harnessing Adversarial Examples*. ICLR (2015). [URL](https://arxiv.org/pdf/1412.6572.pdf).
  
  [2] Pouya Samangouei, Maya Kabkab, Rama Chellappa. *Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models*. ICLR (2018). [URL](https://arxiv.org/pdf/1805.06605.pdf).
  
  
  ---
  
  