Answers.Body: <p>Because the examples are fit to a particular ML model and if you train using different parameters they probably won't be valid.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-03T20:48:27.820Z
Answers.Text: actually, the point of the Goodfellow's talk, that they're very general, what is perturbation for ResNet50, most probably, would be perturbation for VGG16
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-03T20:22:40.063Z
Answers.LastActivityDate: 2021-07-03T20:22:40.063Z
Answers.UserDisplayName: FourierFlux
Answers.Body: <p>I think the problem is that this type of attack will only work for the model that was used to produce the perturbations. These perturbations are computed by backpropagating an error for an image of, say, a panda, but with the true label &quot;airplane&quot;.</p>
  <p>In other words, perturbations are nothing more than gradients indicating in which direction each pixel needs to be changed to make the panda look like an airplane for that particular model. Since the same model will have different weights after each training, this attack will only work for the model used to generate the gradients.</p>
  <p>Here is an illustrative example of this idea when training a generator in a GAN:</p>
  <p><a href="https://i.sstatic.net/Bfdcj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Bfdcj.png" alt="enter image description here" /></a></p>
  <h2>Update</h2>
  <p>While we can transfer an adversarial attack from one model to another, this is only possible under strict constraints. To successfully generate perturbations for the target model, we first need to know the dataset that was used to train it. We also need to know the architecture including the activation and loss functions as well as the hyperparameters of this model. Here is a <a href="https://arxiv.org/abs/1809.02861v4" rel="nofollow noreferrer">work</a> in which the authors take a closer look at this topic.</p>
  <p>Even though it is possible, in my opinion, using CAPTCHAs does not make sense as these attacks may not work in the real world. For example, if we apply this attack to a road sign to trick the autopilot in a vehicle, the lighting conditions and camera perspective can significantly affect the classification.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-04T07:34:21.673Z
Answers.Text: Thanks for a nice pic. However, if these perturbations worked only for a single model, and have no impact on others, this would make use of these captchas, meaningless. However, when training an ensemble of models, one can develop rather universal perturbations - table 4 from https://arxiv.org/pdf/1611.02770.pdf, that are capable of fooling some other model. This work is not very new, so it seems for me to be reasonable to combine EfficientNets, NFNets, ViTs, CaiTs and this would produce reliable and universal adversarial perturbations.
Answers.UserDisplayName: spiridon_the_sun_rotator
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-04T11:20:31.997Z
Answers.Text: Thanks for pointing me to this work. But I still don't believe it will work in practice. The authors state that they are unaware of the dataset being used, but I suspect that the target platform (at least in part) used publicly available datasets like ImageNet as well as pre-trained models. I've updated my answer accordingly. Anyway, this topic is very interesting.
Answers.UserDisplayName: Aray Karjauv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-07-03T21:27:07.830Z
Answers.LastActivityDate: 2021-07-04T11:12:00.157Z
Answers.LastEditDate: 2021-07-04T11:12:00.157Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>In order to check, whether the visitor of the page is a human, and not an AI many web pages and applications have a checking procedure, known as CAPTCHA. These tasks are intended to be simple for people, but unsolvable for machines.</p>
  <p>However, often some text recognition challenges are difficult, like discerning badly, overlapping digits, or telling whether the bus is on the captcha.</p>
  <p>As far as I understand, so far, robustness against adversarial attacks is an unsolved problem. Moreover, adversarial perturbations are rather generalizable and transferrable to various architectures (according to
  <a href="https://youtu.be/CIfsB_EYsVI?t=3226" rel="nofollow noreferrer">https://youtu.be/CIfsB_EYsVI?t=3226</a>).
  This phenomenon is relevant not only to DNN but for simpler linear models.</p>
  <p>With the current state of affairs, it seems to be a good idea, to make CAPTCHAs from these adversarial examples, and the classification problem would be simple for human, without the need to make several attempts to pass this test, but hard for AI.</p>
  <p>There is some research in this field and proposed solutions, but they seem not to be very popular.</p>
  <p>Are there some other problems with this approach, or the owners of the websites (applications) prefer not to rely on this approach?</p>
  
Tags: <computer-vision><adversarial-ml><captcha>
Title: Why adversarial images are not the mainstream for captchas?
UserDisplayName: spiridon_the_sun_rotator
ThreadText: # Why adversarial images are not the mainstream for captchas?
  
  (Asked by: username_0 on 2021-07-03)
  
  In order to check, whether the visitor of the page is a human, and not an AI many web pages and applications have a checking procedure, known as CAPTCHA. These tasks are intended to be simple for people, but unsolvable for machines.
  
  However, often some text recognition challenges are difficult, like discerning badly, overlapping digits, or telling whether the bus is on the captcha.
  
  As far as I understand, so far, robustness against adversarial attacks is an unsolved problem. Moreover, adversarial perturbations are rather generalizable and transferrable to various architectures (according to
  <https://youtu.be/CIfsB_EYsVI?t=3226>).
  This phenomenon is relevant not only to DNN but for simpler linear models.
  
  With the current state of affairs, it seems to be a good idea, to make CAPTCHAs from these adversarial examples, and the classification problem would be simple for human, without the need to make several attempts to pass this test, but hard for AI.
  
  There is some research in this field and proposed solutions, but they seem not to be very popular.
  
  Are there some other problems with this approach, or the owners of the websites (applications) prefer not to rely on this approach?
  
  **Tags:** computer-vision, adversarial-ml, captcha
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-07-03. Score: 2 | Accepted answer
  
  I think the problem is that this type of attack will only work for the model that was used to produce the perturbations. These perturbations are computed by backpropagating an error for an image of, say, a panda, but with the true label "airplane".
  
  In other words, perturbations are nothing more than gradients indicating in which direction each pixel needs to be changed to make the panda look like an airplane for that particular model. Since the same model will have different weights after each training, this attack will only work for the model used to generate the gradients.
  
  Here is an illustrative example of this idea when training a generator in a GAN:
  
  [![enter image description here](https://i.sstatic.net/Bfdcj.png)](https://i.sstatic.net/Bfdcj.png)
  
  ## Update
  
  While we can transfer an adversarial attack from one model to another, this is only possible under strict constraints. To successfully generate perturbations for the target model, we first need to know the dataset that was used to train it. We also need to know the architecture including the activation and loss functions as well as the hyperparameters of this model. Here is a [work](https://arxiv.org/abs/1809.02861v4) in which the authors take a closer look at this topic.
  
  Even though it is possible, in my opinion, using CAPTCHAs does not make sense as these attacks may not work in the real world. For example, if we apply this attack to a road sign to trick the autopilot in a vehicle, the lighting conditions and camera perspective can significantly affect the classification.
  
  > **Comments:**
  > **username_0**: Thanks for a nice pic. However, if these perturbations worked only for a single model, and have no impact on others, this would make use of these captchas, meaningless. However, when training an ensemble of models, one can develop rather universal perturbations - table 4 from https://arxiv.org/pdf/1611.02770.pdf, that are capable of fooling some other model. This work is not very new, so it seems for me to be reasonable to combine EfficientNets, NFNets, ViTs, CaiTs and this would produce reliable and universal adversarial perturbations.  
  > **username_2**: Thanks for pointing me to this work. But I still don't believe it will work in practice. The authors state that they are unaware of the dataset being used, but I suspect that the target platform (at least in part) used publicly available datasets like ImageNet as well as pre-trained models. I've updated my answer accordingly. Anyway, this topic is very interesting.  
  
  ---
  
  ### Answer by username_1 on 2021-07-03. Score: 0
  
  Because the examples are fit to a particular ML model and if you train using different parameters they probably won't be valid.
  
  > **Comments:**
  > **username_0**: actually, the point of the Goodfellow's talk, that they're very general, what is perturbation for ResNet50, most probably, would be perturbation for VGG16  
  
  ---
  
  