Answers.Body: <blockquote>
  <p><a href="https://arxiv.org/abs/1312.6199" rel="nofollow noreferrer">Adversarial examples</a> are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they’re like optical illusions for machines.</p>
  </blockquote>
  <p>Source: <a href="https://blog.openai.com/adversarial-example-research/" rel="nofollow noreferrer">Attacking Machine Learning with Adversarial Examples</a></p>
  <p>You create an input and test against the output, tuning the input to maximize the error. There are different criteria for tuning the input, sometimes you might want the shortest possible input to create the largest error, often you would want very similar input to cause the greatest error.</p>
  <p>Example: I take a \$1 bill and write a few more zeros after the one; you accept it as a \$1000 bill.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-31T07:41:51.717Z
Answers.LastActivityDate: 2021-12-10T19:05:31.723Z
Answers.LastEditDate: 2021-12-10T19:05:31.723Z
Answers.UserDisplayName: Rob
Answers.Body: <p>I guess they are talking about adversarial attacks in the same way that Szegedy et al. did in <a href="https://arxiv.org/abs/1312.6199" rel="nofollow noreferrer">&quot;Intriguing properties of neural networks&quot;</a></p>
  <p>They described &quot;adversarial attacks&quot; or &quot;adversarial examples&quot; as images with hardly perceptible perturbations that change the network's prediction.</p>
  <p>For example, imagine you've trained a CNN to classify between a variety of classes. You take a picture of a dog <span class="math-container">$X_1$</span>, and your CNN correctly classified it as a &quot;dog&quot;, everything is fine so far.</p>
  <p>Then you can add some small perturbation <span class="math-container">$p$</span> to your image <span class="math-container">$X_1$</span>, so now you have a new image <span class="math-container">$X_2 = X_1 + p$</span>. This new image still looks like a dog, because your perturbation was so small that is almost imperceptible.</p>
  <p>The problem is that your CNN will classify your picture <span class="math-container">$X_2$</span> as something that is not a dog, for example, &quot;fish&quot;.</p>
  <p>Here, <span class="math-container">$X_2$</span> is an <strong>adversarial example</strong> created after using an adversarial perturbation <span class="math-container">$p$</span>.</p>
  <p>What is interesting about these adversarial perturbations <span class="math-container">$p$</span> is that they are not random. Actually, CNNs are very robust to random perturbations (noise), but adversarial perturbations <span class="math-container">$p$</span> are not like them. They are computed to fool a classifier (not only CNNs).</p>
  <p>You can refer to figure 5 of the aforementioned paper for more examples.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-16T23:00:42.540Z
Answers.LastActivityDate: 2021-12-10T19:08:11.837Z
Answers.LastEditDate: 2021-12-10T19:08:11.837Z
Answers.UserDisplayName: Ita
Answers.Body: <p>Simple answer is tweaking an image in unnoticeable ways that completely fool software. Eg a cat that is identified as 99% likely "to be guacamole" <a href="https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program/#CU7dSAfQ5sqY" rel="nofollow noreferrer">https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program/#CU7dSAfQ5sqY</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-01-03T00:17:04.123Z
Answers.LastActivityDate: 2019-01-03T00:17:04.123Z
Answers.UserDisplayName: Julian H
Body: <p>I'm reading this really interesting article <a href="https://arxiv.org/pdf/1712.02950.pdf" rel="nofollow noreferrer">CycleGAN, a Master of Steganography</a>. I understand everything up until this paragraph:</p>
  <blockquote>
  <p>we may view the CycleGAN training procedure as continually mounting an <strong>adversarial attack</strong> on <span class="math-container">$G$</span>, by optimizing a generator <span class="math-container">$F$</span> to generate adversarial maps that force <span class="math-container">$G$</span> to produce a desired image. Since we have demonstrated that it is possible to generate these adversarial maps using gradient descent, it is nearly certain that the training procedure is also causing <span class="math-container">$F$</span> to generate these adversarial maps. As <span class="math-container">$G$</span> is also being optimized, however, <span class="math-container">$G$</span> may actually be seen as cooperating in this attack by learning to become increasingly susceptible to attacks. We observe that the magnitude of the difference <span class="math-container">$y^{*}-y_{0}$</span> necessary to generate a convincing adversarial example by Equation 3 decreases as the CycleGAN model trains, indicating cooperation of <span class="math-container">$G$</span> to support adversarial maps.</p>
  </blockquote>
  <p>How is the CycleGAN training procedure an adversarial attack?</p>
  <p>I don't really understand the quoted explanation.</p>
  
Tags: <terminology><papers><adversarial-ml><adversarial-attacks><cycle-gan>
Title: What is an adversarial attack?
UserDisplayName: Cyclist
ThreadText: # What is an adversarial attack?
  
  (Asked by: username_0 on 2018-12-31)
  
  I'm reading this really interesting article [CycleGAN, a Master of Steganography](https://arxiv.org/pdf/1712.02950.pdf). I understand everything up until this paragraph:
  
  > we may view the CycleGAN training procedure as continually mounting an **adversarial attack** on $G$, by optimizing a generator $F$ to generate adversarial maps that force $G$ to produce a desired image. Since we have demonstrated that it is possible to generate these adversarial maps using gradient descent, it is nearly certain that the training procedure is also causing $F$ to generate these adversarial maps. As $G$ is also being optimized, however, $G$ may actually be seen as cooperating in this attack by learning to become increasingly susceptible to attacks. We observe that the magnitude of the difference $y^{\*}-y\_{0}$ necessary to generate a convincing adversarial example by Equation 3 decreases as the CycleGAN model trains, indicating cooperation of $G$ to support adversarial maps.
  
  How is the CycleGAN training procedure an adversarial attack?
  
  I don't really understand the quoted explanation.
  
  **Tags:** terminology, papers, adversarial-ml, adversarial-attacks, cycle-gan
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2018-12-31. Score: 2
  
  > [Adversarial examples](https://arxiv.org/abs/1312.6199) are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake; they’re like optical illusions for machines.
  
  Source: [Attacking Machine Learning with Adversarial Examples](https://blog.openai.com/adversarial-example-research/)
  
  You create an input and test against the output, tuning the input to maximize the error. There are different criteria for tuning the input, sometimes you might want the shortest possible input to create the largest error, often you would want very similar input to cause the greatest error.
  
  Example: I take a \$1 bill and write a few more zeros after the one; you accept it as a \$1000 bill.
  
  
  ---
  
  ### Answer by username_2 on 2020-01-16. Score: 1
  
  I guess they are talking about adversarial attacks in the same way that Szegedy et al. did in ["Intriguing properties of neural networks"](https://arxiv.org/abs/1312.6199)
  
  They described "adversarial attacks" or "adversarial examples" as images with hardly perceptible perturbations that change the network's prediction.
  
  For example, imagine you've trained a CNN to classify between a variety of classes. You take a picture of a dog $X\_1$, and your CNN correctly classified it as a "dog", everything is fine so far.
  
  Then you can add some small perturbation $p$ to your image $X\_1$, so now you have a new image $X\_2 = X\_1 + p$. This new image still looks like a dog, because your perturbation was so small that is almost imperceptible.
  
  The problem is that your CNN will classify your picture $X\_2$ as something that is not a dog, for example, "fish".
  
  Here, $X\_2$ is an **adversarial example** created after using an adversarial perturbation $p$.
  
  What is interesting about these adversarial perturbations $p$ is that they are not random. Actually, CNNs are very robust to random perturbations (noise), but adversarial perturbations $p$ are not like them. They are computed to fool a classifier (not only CNNs).
  
  You can refer to figure 5 of the aforementioned paper for more examples.
  
  
  ---
  
  ### Answer by username_3 on 2019-01-03. Score: 0
  
  Simple answer is tweaking an image in unnoticeable ways that completely fool software. Eg a cat that is identified as 99% likely "to be guacamole" <https://mashable.com/2017/11/02/mit-researchers-fool-google-ai-program/#CU7dSAfQ5sqY>
  
  
  ---
  
  