Answers.Body: <p>As far as I know, there are only hypothesis on this (<a href="https://simons.berkeley.edu/talks/new-perspective-adversarial-perturbations" rel="nofollow noreferrer">https://simons.berkeley.edu/talks/new-perspective-adversarial-perturbations</a>) which pretty much relies on the idea that there are &quot;spurious&quot; correlation in the dataset used for training (check the video in the link for more informations, they have also some papers referenced to read more about it)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-22T20:48:21.897Z
Answers.LastActivityDate: 2024-02-22T20:48:21.897Z
Answers.UserDisplayName: Alberto
Body: <p>I have read (*) that a common technique to attack a <em>black box</em> AI system based on a neural network is to use it to train a <em>surrogate model</em> to make the same classifications as the black box one.</p>
  <p>Once this is done, one can look for adversarial examples on the surrogate model (on which the attacker has access to all the weights and can compute gradients).</p>
  <p>The key property that makes such attacks successful is <em>transferability</em>: an adversarial example on the surrogate model is likely to be an adversarial example on the black box model.</p>
  <p><strong>Question:</strong> Do we know why such transferability properties hold and under which conditions?</p>
  <p>(*) Although I don't remember where I first read it, this is mentioned in the book <a href="https://www.goodreads.com/book/show/78294325-not-with-a-bug-but-with-a-sticker" rel="nofollow noreferrer">Not with a bug but with a sticker</a> although it doesn't get into technical details (this is a book for the general public).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-22T10:09:47.687Z
Comments.Text: A reference or link to where you have read this information would help a lot for anyone attempting an answer
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-02-22T10:19:14.043Z
Comments.Text: @NeilSlater Right, I added a reference in my question.
Comments.UserDisplayName: Weier
Tags: <deep-learning><adversarial-ml><ai-security><adversarial-attacks>
Title: Why do adversarial attack transfer well?
UserDisplayName: Weier
ThreadText: # Why do adversarial attack transfer well?
  
  (Asked by: username_0 on 2024-02-22)
  
  I have read (\*) that a common technique to attack a *black box* AI system based on a neural network is to use it to train a *surrogate model* to make the same classifications as the black box one.
  
  Once this is done, one can look for adversarial examples on the surrogate model (on which the attacker has access to all the weights and can compute gradients).
  
  The key property that makes such attacks successful is *transferability*: an adversarial example on the surrogate model is likely to be an adversarial example on the black box model.
  
  **Question:** Do we know why such transferability properties hold and under which conditions?
  
  (\*) Although I don't remember where I first read it, this is mentioned in the book [Not with a bug but with a sticker](https://www.goodreads.com/book/show/78294325-not-with-a-bug-but-with-a-sticker) although it doesn't get into technical details (this is a book for the general public).
  
  **Tags:** deep-learning, adversarial-ml, ai-security, adversarial-attacks
  > **Comments:**
  > **username_1**: A reference or link to where you have read this information would help a lot for anyone attempting an answer  
  > **username_0**: @NeilSlater Right, I added a reference in my question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2024-02-22. Score: 0
  
  As far as I know, there are only hypothesis on this (<https://simons.berkeley.edu/talks/new-perspective-adversarial-perturbations>) which pretty much relies on the idea that there are "spurious" correlation in the dataset used for training (check the video in the link for more informations, they have also some papers referenced to read more about it)
  
  
  ---
  
  