Answers.Body: <p>The images that you provided may be unrecognizable for us. They are actually the images that we recognize but evolved using the <a href="https://github.com/sferes2/sferes2" rel="noreferrer">Sferes</a> evolutionary framework.</p>
  <p>While these images are almost impossible for humans to label with anything but abstract arts, the Deep Neural Network will label them to be familiar objects with 99.99% confidence.</p>
  <p>This result highlights differences between how DNNs and humans recognize objects. Images are either directly (or indirectly) encoded</p>
  <p>According to this <a href="https://youtu.be/M2IebCN9Ht4" rel="noreferrer">video</a></p>
  <blockquote>
  <p>Changing an image originally correctly classified in a way imperceptible to humans can cause the cause DNN to classify it as something else.</p>
  <p>In the image below the number at the bottom are the images are supposed to look like the digits
  But the network believes the images at the top (the one like white noise) are real digits with 99.99% certainty.</p>
  </blockquote>
  <p><img src="https://i.sstatic.net/Jx1wX.png" alt="enter image description here" /></p>
  <blockquote>
  <p>The main reason why these are easily fooled is that Deep Neural Network does not see the world in the same way as human vision. We use the whole image to identify things while DNN depends on the features. As long as DNN detects certain features, it will classify the image as a familiar object it has been trained on.
  The researchers proposed one way to prevent such fooling by adding the fooling images to the dataset in a new class and training DNN on the enlarged dataset. In the experiment, the confidence score decreases significantly for ImageNet AlexNet. It is not easy to fool the retrained DNN this time. But when the researchers applied such method to MNIST LeNet, evolution still produces many unrecognizable images with confidence scores of 99.99%.</p>
  </blockquote>
  <p>More details <a href="http://www.evolvingai.org/fooling" rel="noreferrer">here</a> and <a href="http://www.kdnuggets.com/2015/01/deep-learning-can-be-easily-fooled.html" rel="noreferrer">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-06T17:41:35.647Z
Answers.LastActivityDate: 2018-04-13T03:22:19.983Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Vishnu JK
Answers.Body: <p>All answers here are great, but, for some reason, nothing has been said so far on why this effect <em>should not surprise</em> you. I'll fill the blank.</p>
  
  <p>Let me start with one requirement that is absolutely essential for this to work: the attacker <em>must know neural network architecture</em> (number of layers, size of each layer, etc). Moreover, in all cases that I examined myself, the attacker knows the snapshot of the model that is used in production, i.e. all weights. In other words, the "source code" of the network isn't a secret.</p>
  
  <p>You can't fool a neural network if you treat it like a black box. And you can't reuse the same fooling image for different networks. 
  In fact, you have to "train" the target network yourself, and here by training I mean to run forward and backprop passes, but specially crafted for another purpose.</p>
  
  <h3>Why is it working at all?</h3>
  
  <p>Now, here's the intuition. Images are very high dimensional: even the space of small 32x32 color images has <code>3 * 32 * 32 = 3072</code> dimensions. But the training data set is relatively small and contains real pictures, all of which have some structure and nice statistical properties (e.g. smoothness of color). So the training data set is located on a tiny manifold of this huge space of images. </p>
  
  <p>The convolutional networks work extremely well on this manifold, but basically, know nothing about the rest of the space. The classification of the points outside of the manifold is just a linear extrapolation based on the points inside the manifold. No wonder that some particular points are extrapolated incorrectly. The attacker only needs a way to navigate to the closest of these points.</p>
  
  <h3>Example</h3>
  
  <p>Let me give you a concrete example how to fool a neural network. To make it compact, I'm going to use a very simple logistic regression network with one nonlinearity (sigmoid). It takes a 10-dimensional input <code>x</code>, computes a single number <code>p=sigmoid(W.dot(x))</code>, which is the probability of class 1 (versus class 0).</p>
  
  <p><img src="https://i.sstatic.net/i3mO7.png" alt="logistic regression"></p>
  
  <p>Suppose you know <code>W=(-1, -1, 1, -1, 1, -1, 1, 1, -1, 1)</code> and start with an input <code>x=(2, -1, 3, -2, 2, 2, 1, -4, 5, 1)</code>. A forward pass gives <code>sigmoid(W.dot(x))=0.0474</code> or 95% probability that <code>x</code> is class 0 example.  </p>
  
  <p><img src="https://i.sstatic.net/V6cNn.png" alt="x and W"></p>
  
  <p>We'd like to find another example, <code>y</code>, which is very close to <code>x</code> but is classified by the network as 1. Note that <code>x</code> is 10-dimensional, so we have the freedom to nudge 10 values, which is a lot. </p>
  
  <p>Since <code>W[0]=-1</code> is negative, it's better for to have a small <code>y[0]</code> to make a total contribution of <code>y[0]*W[0]</code> small. Hence, let's make <code>y[0]=x[0]-0.5=1.5</code>.
  Likewise, <code>W[2]=1</code> is positive, so it's better to increase <code>y[2]</code> to make <code>y[2]*W[2]</code> bigger: <code>y[2]=x[2]+0.5=3.5</code>. And so on.</p>
  
  <p><img src="https://i.sstatic.net/8zs12.png" alt="x, W and y"></p>
  
  <p>The result is <code>y=(1.5, -1.5, 3.5, -2.5, 2.5, 1.5, 1.5, -3.5, 4.5, 1.5)</code>, and <code>sigmoid(W.dot(y))=0.88</code>. With this one change we improved the class 1 probability from 5% to 88%!</p>
  
  <h3>Generalization</h3>
  
  <p>If you look closely at the previous example, you'll notice that I knew exactly how to tweak <code>x</code> in order to move it to the target class, because I knew the network gradient. What I did was actually a <em>backpropagation</em>, but with respect to the data, instead of weights.</p>
  
  <p>In general, the attacker starts with target distribution <code>(0, 0, ..., 1, 0, ..., 0)</code> (zero everywhere, except for the class it wants to achieve), backpropagates to the data and makes a tiny move in that direction. Network state is not updated.</p>
  
  <p>Now it should be clear that it's a common feature of feed-forward networks that deal with a small data manifold, no matter how deep it is or the nature of data (image, audio, video or text).</p>
  
  <h3>Potection</h3>
  
  <p>The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request.
  It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easily mess up
  with the results obtained for another network. The more networks are used, the more complex an attack becomes.</p>
  
  <p>Another possibility is to smooth the input before passing it to the network.</p>
  
  <h3>Positive use of the same idea</h3>
  
  <p>You shouldn't think that backpropagation to the image has only negative applications. A very similar technique, called <em>deconvolution</em>, is used for visualization and better understanding what neurons have learned.</p>
  
  <p>This technique allows synthesizing an image that causes a particular neuron to fire, basically see visually "what the neuron is looking for", which in general makes convolutional neural networks more interpretable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T17:08:44.780Z
Answers.Text: Reading the paper it seems that the recognizer was treated as a black box.
Answers.UserDisplayName: George White
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-30T03:14:38.107Z
Answers.Text: If @GeorgeWhite is correct then this invalidates this otherwise great-looking answer. Do you have any comment on that?
Answers.UserDisplayName: Caleb Stanford
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-18T14:30:02.917Z
Answers.LastActivityDate: 2018-04-12T14:53:26.000Z
Answers.LastEditDate: 2018-04-12T14:53:26.000Z
Answers.UserDisplayName: Maxim
Answers.Body: <p>An important question that does not yet have a satisfactory answer in neural network research is how DNNs come up with the predictions they offer. DNNs effectively work (though not exactly) by matching patches in the images to a "dictionary" of patches, one stored in each neuron (see <a href="http://research.google.com/archive/unsupervised_icml2012.html">the youtube cat paper</a>). Thus, it may not have a high level view of the image since it only looks at patches, and images are usually downscaled to much lower resolution to obtain the results in current generation systems. Methods which look at how the components of the image interact may be able to avoid these problems.</p>
  
  <p>Some questions to ask for this work are: How confident were the networks when they made these predictions? How much volume do such adversarial images occupy in the space of all images?</p>
  
  <p>Some work I am aware of in this regard comes from Dhruv Batra and Devi Parikh's Lab at Virginia Tech who look into this for question answering systems: <a href="http://arxiv.org/pdf/1606.07356v1.pdf">Analyzing the Behavior of Visual Question Answering Models</a> and <a href="https://filebox.ece.vt.edu/~dbatra/papers/gmpb_icmlvis16.pdf">Interpreting Visual Question Answering models</a>. </p>
  
  <p>More such work is needed, and just as the human visual system does also get fooled by such "optical illusions", these problems may be unavoidable if we use DNNs, though AFAIK nothing is yet known either way, theoretically or empirically. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T23:25:53.977Z
Answers.LastActivityDate: 2016-08-03T23:25:53.977Z
Answers.UserDisplayName: Harsh
Answers.Body: <blockquote>
  <p>How is it possible that deep neural networks are so easily fooled?</p>
  <p>Deep neural networks are easily fooled by giving high confidence predictions for unrecognizable images. How is this possible? Can you please explain ideally in <strong>plain</strong> English?</p>
  </blockquote>
  <p>Intuitively, extra hidden layers ought to make the network able to learn more complex classification functions, and thus do a better job classifying. While it may be named <em>deep learning</em> it's actually a shallow understanding.</p>
  <p>Test your own knowledge: Which animal in the grid below is a <a href="https://en.wikipedia.org/wiki/Cat" rel="noreferrer">Felis silvestris catus</a>, take your time and no cheating. Here's a hint: which is a domestic house cat?</p>
  <p><a href="https://i.sstatic.net/P09Tk.jpg" rel="noreferrer"><img src="https://i.sstatic.net/P09Tk.jpg" alt="Which is a domestic house cat?" /></a></p>
  <p>For a better understanding checkout: &quot;<a href="https://medium.com/visweekly/adversarial-attack-to-vulnerable-visualizations-bfa3ed50efc2" rel="noreferrer">Adversarial Attack to Vulnerable Visualizations</a>&quot; <strong>and</strong> &quot;<a href="http://neuralnetworksanddeeplearning.com/chap5.html" rel="noreferrer">Why are deep neural networks hard to train?</a>&quot;.</p>
  <p>The problem is analogous to <a href="https://en.wikipedia.org/wiki/Aliasing" rel="noreferrer">aliasing</a>, an effect that causes different signals to become indistinguishable (or aliases of one another) when sampled, and the <a href="https://en.m.wikipedia.org/wiki/Wagon-wheel_effect" rel="noreferrer">stagecoach-wheel effect</a>, where a spoked wheel appears to rotate differently from its true rotation.</p>
  <p>The neural network doesn't <em>know</em> what it's looking at or which way it's going.</p>
  <p>Deep neural networks aren't an expert on something, they are trained to decide mathematically that some goal has been met, if they are not trained to reject wrong answers they don't have a concept of what is wrong; they only <em>know</em> what is correct and what is not correct - wrong and &quot;not correct&quot; are not necessarily the same thing, neither is &quot;correct&quot; and true.</p>
  <p>The neural network doesn't <em>know</em> right from wrong.</p>
  <p>Just like most people wouldn't know a house cat if they saw one, two or more, or none. How many house cats in the above photo grid, none. Any accusations of including cute cat pictures is unfounded, those are all dangerous wild animals.</p>
  <p>Here's another example. Does answering the question make Bart and Lisa smarter, does the person they are asking even know, are there unknown variables that can come into play?</p>
  <p><a href="https://i.sstatic.net/BznDK.jpg" rel="noreferrer"><img src="https://i.sstatic.net/BznDK.jpg" alt="Are we there yet?" /></a></p>
  <p>We aren't there yet but neural networks can quickly provide an answer that is likely to be correct, especially if it was properly trained to avoid all misteps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-06T04:55:18.760Z
Answers.LastActivityDate: 2018-09-06T05:01:12.717Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Rob
Answers.Body: <p>Can't comment(due to that required 50 rep), but I wanted to make a response to Vishnu JK and the OP. I think you guys are skipping the fact that the neural network only really is saying truly from a programmatic standpoint that "this is most like".</p>
  
  <p>For example, while we can list the above image examples as "abstract art", they definitively are most like was is listed. Remember learning algorithms have a scope on what they recognize as an object and if you look at all the above examples... and think about the scope of the algorithum... these make sense (even the ones at a glance we would recognize as white noise). In Vishnu example of the numbers, if you fuzz your eyes and bring the images out of focus, you can actually in every case spot patterns that really closely reflect the numbers in question.</p>
  
  <p>The problem that is being shown here is that the algorithm appears to not have a "unknown case". Basically when the pattern recognition says that it doesn't exist in the output scope. (so a final output node group that says this is nothing that I know off). For example, people do this as well, as it's one thing humans and learning algorithms have in common. Here's a link to show what I'm talking about (what is the following, define it) using only known animals that exist:</p>
  
  <p><img src="https://i.sstatic.net/AZwnpm.jpg" alt="Picture link"></p>
  
  <p>Now as a person, limited by what I know and can say, I'd have to conclude that the following is an elephant. But it's not. Learning algorithms (for the most part) do not have a "like a" statement, the out put always validates down to a confidence percentage. So tricking one in this fashion is not surprising... what is of course surprising is that based on it's knowledge set, it actually comes to the point in which, if you look at the above cases listed by OP and Vishnu that a person... with a little looking... can see how the learning algorithm probable made the association. </p>
  
  <p>So, I wouldn't really call it a mislabel on the part of the algorithm, or even call it a case where it's been tricked... rather a case where it's scope was developed incorrectly.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T17:11:08.487Z
Answers.Text: You are forgetting the 99% confidence level the researchers found.
Answers.UserDisplayName: George White
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-31T10:48:38.523Z
Answers.Text: Indeed, an unknown image should be expected to return a low confidence level in every category. Is it a cat? No. Is it a dog? No. Is it a house? No. Is it a calculator? Sort of, it has all these buttons in a grid, but there's no screen and no writing on the buttons, so no.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-10-14T17:29:22.983Z
Answers.LastActivityDate: 2016-10-14T17:31:31.923Z
Answers.LastEditDate: 2016-10-14T17:31:31.923Z
Answers.UserDisplayName: Metric
Answers.Body: <p>The neural networks can be easily fooled or hacked by adding certain structured noise in image space (<a href="https://arxiv.org/abs/1312.6199" rel="nofollow">Szegedy 2013</a>, <a href="http://arxiv.org/abs/1412.1897" rel="nofollow">Nguyen 2014</a>) due to ignoring non-discriminative information in their input.</p>
  
  <p>For example:</p>
  
  <blockquote>
    <p>Learning to detect jaguars by matching the unique spots on their fur while ignoring the fact that they have four legs.<sup><a href="http://arxiv.org/abs/1506.06579" rel="nofollow">2015</a></sup></p>
  </blockquote>
  
  <p>So basically the high confidence prediction in certain models exists due to a '<em>combination of their locally linear nature and high-dimensional input space</em>'.<sup><a href="http://arxiv.org/abs/1412.1897" rel="nofollow">2015</a></sup></p>
  
  <p>Published as a conference paper at <a href="http://www.stat.ucla.edu/~ywu/ICLR2015.pdf" rel="nofollow">ICLR 2015</a> (work by Dai) suggest that transferring discriminatively trained parameters to generative models, could be a great area for further improvements.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-11T03:08:39.273Z
Answers.LastActivityDate: 2016-08-11T03:08:39.273Z
Answers.UserDisplayName: kenorb
Answers.Body: <p>Neural networks are easily fooled, provided you know how to fool them.</p>
  
  <p>Consider a linear network with an input layer and an output layer, which has an error function E (we don't need hidden layers to show how to fool a network). For a given input image x, E measures the (squared) difference between the network's output y and the desired (correct) output. </p>
  
  <p>The output unit’s state y is given by the inner product of x with the output unit’s weight vector w,so that </p>
  
  <pre><code>y=w·x. 
  </code></pre>
  
  <p>If we change x to x′ by adding ∆x then the output will change by ∆y to</p>
  
  <pre><code>y′ = w·x+w·∆x (9.4) 
  
     = y + ∆y. (9.5)
  </code></pre>
  
  <p>Notice that ∆x defines a direction in the input space; the question is, which direction ∆x will have most impact on y? </p>
  
  <p>By definition, a change in x in the direction ∇E produces the largest
  possible change in y. An adversarial image x' is constructed by taking the derivative ∇E of E with respect to the input image x, that is,  </p>
  
  <pre><code>   x′ = x + ε∇E,
  </code></pre>
  
  <p>where ε is a small constant. 
  By definition, ∇E is the direction of steepest ascent, so the modification ε∇E to x will alter y more than a change in any other direction.</p>
  
  <p><a href="https://i.sstatic.net/ggkFa.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ggkFa.jpg" alt="enter image description here"></a></p>
  
  <p>This is an extract from the book: <a href="https://jim-stone.staff.shef.ac.uk/BookBayes2012/books_by_jv_stone" rel="nofollow noreferrer">Artificial Intelligence Engines:  A Tutorial Introduction to the Mathematics of Deep Learning</a> (2019). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T09:18:36.353Z
Answers.LastActivityDate: 2020-01-24T12:33:02.420Z
Answers.LastEditDate: 2020-01-24T12:33:02.420Z
Answers.UserDisplayName: James V Stone
Answers.Body: <p>There is already many good answers, I will just add to those that came before mine: </p>
  
  <p>This type of images you are referring to are called <a href="http://arxiv.org/pdf/1412.1897" rel="nofollow noreferrer">adversarial perturbations</a>, (see <a href="https://arxiv.org/pdf/1412.1897.pdf" rel="nofollow noreferrer">1</a>, and it is not limited to images, it has been shown to apply to text too, see <a href="https://arxiv.org/abs/1707.07328" rel="nofollow noreferrer">Jia &amp; Liang, EMNLP 2017</a>. In text, the introduction of an irrelevant sentence which doesn't contradict the paragraph has been seen to cause the network to come to a completely different answer (see see <a href="https://arxiv.org/abs/1707.07328" rel="nofollow noreferrer">Jia &amp; Liang, EMNLP 2017</a>). </p>
  
  <p><strong>The reason they work</strong> is due to the fact that neural network view images in a different way from us, coupled with the high dimentionality of the problem space. Where we see the whole picture, they see a combination of features which combine to form an object(<a href="https://arxiv.org/abs/1610.08401" rel="nofollow noreferrer">Moosavi-Dezfooli et al., CVPR 2017</a>). According to  perturbation generated against one network has been seen to have high likelihood to work on other networks: </p>
  
  <p><a href="https://i.sstatic.net/7zK85.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7zK85.jpg" alt="perturbations"></a></p>
  
  <p>In the figure above, it is seen that The universal perturbations computed for the VGG-19 network, for example, have a fooling ratio above 53% for all other tested architectures.</p>
  
  <p>So how do you deal with the threat of adversarial perturbations? 
  Well, for one, you can try to generate as many perturbations as you can and use them to fine-tune your model. Whist this somewhat solves the problem, it doesn't solve the problem entirely. In (<a href="https://arxiv.org/abs/1610.08401" rel="nofollow noreferrer">Moosavi-Dezfooli et al., CVPR 2017</a>) the author reported that, repeating the process by computing new perturbations and then fine-tuning again seems to yield no further improvements, regardless of the number of iterations, with the fooling ratio hovering around 80%.</p>
  
  <p>Perturbations are an indication of the shallow pattern matching that neural networks perform, coupled with their minimal lack of in-depth understanding of the problem at hand. More work still needs to be done. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-12T08:05:51.527Z
Answers.LastActivityDate: 2017-10-12T08:18:30.067Z
Answers.LastEditDate: 2017-10-12T08:18:30.067Z
Answers.UserDisplayName: Tshilidzi Mudau
Answers.Body: <p>First up, those images (even the first few) aren't complete trash despite being junk to humans; they're actually finely tuned with various advanced techniques, including another neural network.</p>
  
  <blockquote>
    <p>The deep neural network is the pre-trained network modeled on AlexNet provided by <a href="https://github.com/BVLC/caffe">Caffe</a>. To evolve images, both the directly encoded and indirectly encoded images, we use the <a href="https://github.com/jbmouret/sferes2">Sferes</a> evolutionary framework. The entire code base to conduct the evolutionary experiments can be download [sic] <a href="https://github.com/Evolving-AI-Lab/fooling">here</a>. The code for the images produced by gradient ascent is available <a href="https://github.com/Evolving-AI-Lab/fooling/tree/master/caffe/ascent">here</a>.</p>
  </blockquote>
  
  <p>Images that are actually random junk were correctly recognized as nothing meaningful:</p>
  
  <blockquote>
    <p>In response to an unrecognizable image, the networks could have output a low confidence for each of the 1000 classes, instead of an extremely high confidence value for one of the classes. In fact, they do just that for randomly generated images (e.g. those in generation 0 of the evolutionary run)</p>
  </blockquote>
  
  <p>The original goal of the researchers was to use the neural networks to automatically generate images that look like the real things (by getting the recognizer's feedback and trying to change the image to get a more confident result), but they ended up creating the above art. Notice how even in the static-like images there are little splotches - usually near the center - which, it's fair to say, are triggering the recognition.</p>
  
  <blockquote>
    <p>We were not trying to produce adversarial, unrecognizable images. Instead, we were trying to produce recognizable images, but these unrecognizable images emerged.</p>
  </blockquote>
  
  <p>Evidently, these images had just the right distinguishing features to match what the AI looked for in pictures. The "paddle" image does have a paddle-like shape, the "bagel" is round and the right color, the "projector" image is a camera-lens-like thing, the "computer keyboard" is a bunch of rectangles (like the individual keys), and the "chainlink fence" legitimately looks like a chain-link fence to me.</p>
  
  <blockquote>
    <p>Figure 8. Evolving images to match DNN classes produces a tremendous diversity of images. Shown are images selected to showcase diversity from 5 evolutionary runs. The diversity suggests that the images are non-random, but that instead evolutions producing [sic] discriminative features of each target class.</p>
  </blockquote>
  
  <p>Further reading: <a href="http://www.evolvingai.org/files/DNNsEasilyFooled_cvpr15.pdf">the original paper</a> (large PDF)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T17:07:28.467Z
Answers.Text: They were not forced to put it in 1 of a 1000 categories. They gave 99% confidence of 1 category.
Answers.UserDisplayName: George White
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-09T17:55:35.163Z
Answers.Text: The problem is - How DNN gives such high confidence ~ `99% !` that these some-object-pattern-like pictures are representing true objects ? Unless DNN is mentally ill and thinks it takes [Rorschach test](https://en.wikipedia.org/wiki/Rorschach_test) in some hospital :-)
Answers.UserDisplayName: Agnius Vasiliauskas
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-01-10T08:55:53.990Z
Answers.Text: Sorry, I couldn't resist :-) "Bagel" is round, but so it is sun,eyeball,etc.
  "Projector" could be an atom model, laser beam cross section, etc. And so on and so on. So why the hell, bagel is preferred over sun, projector is preferred over atom schema, etc. with `99%` confidence ? For me - the main difference between DNN and human recognition is the fact that humans are not forced to recognize something, while NN seems are !
Answers.UserDisplayName: Agnius Vasiliauskas
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-03T18:18:58.077Z
Answers.LastActivityDate: 2016-08-03T18:18:58.077Z
Answers.UserDisplayName: Ben N
Body: <p>The following <a href="http://www.evolvingai.org/fooling" rel="noreferrer">page</a>/<a href="http://www.evolvingai.org/files/DNNsEasilyFooled_cvpr15.pdf" rel="noreferrer">study</a> demonstrates that the deep neural networks are easily fooled by giving high confidence predictions for unrecognisable images, e.g.</p>
  
  <p><a href="https://i.sstatic.net/7pgrH.jpg" rel="noreferrer"><img src="https://i.sstatic.net/7pgrH.jpg" alt="Evolved images that are unrecognisable to humans, but that state-of-the-art DNNs trained on ImageNet believe with &gt;= 99.6% certainty to be a familiar object. This result highlights differences between how DNNs and humans recognise objects. Directly and indirectly encoded images"></a></p>
  
  <p><a href="https://i.sstatic.net/pBm48.png" rel="noreferrer"><img src="https://i.sstatic.net/pBm48.png" alt="Evolving images to match DNN classes produces a tremendous diversity of images. The mean DNN confidence scores for these images is 99.12% for the listed class, meaning that the DNN believes with near-certainty that the image is that type of thing. Shown are images selected to showcase diversity from 5 independent evolutionary runs. The images shed light on what the DNN network cares about, and what it does not, when classifying an image. For example, a school bus is alternating yellow and black lines, but does not need to have a windshield or wheels"></a></p>
  
  <p>How this is possible? Can you please explain ideally in plain English?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-05T22:50:14.007Z
Comments.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105225/discussion-on-question-by-kenorb-how-is-it-possible-that-deep-neural-networks-ar).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-22T21:37:43.990Z
Comments.Text: Related video: ['How neural networks learn' - Part II: Adversarial Examples](https://www.youtube.com/watch?v=4rFOkpI0Lcg).
Comments.UserDisplayName: kenorb
Tags: <convolutional-neural-networks><computer-vision><image-recognition><deep-neural-networks><adversarial-ml>
Title: How is it possible that deep neural networks are so easily fooled?
UserDisplayName: kenorb
ThreadText: # How is it possible that deep neural networks are so easily fooled?
  
  (Asked by: username_0 on 2016-08-02)
  
  The following [page](http://www.evolvingai.org/fooling)/[study](http://www.evolvingai.org/files/DNNsEasilyFooled_cvpr15.pdf) demonstrates that the deep neural networks are easily fooled by giving high confidence predictions for unrecognisable images, e.g.
  
  [![Evolved images that are unrecognisable to humans, but that state-of-the-art DNNs trained on ImageNet believe with >= 99.6% certainty to be a familiar object. This result highlights differences between how DNNs and humans recognise objects. Directly and indirectly encoded images](https://i.sstatic.net/7pgrH.jpg)](https://i.sstatic.net/7pgrH.jpg)
  
  [![Evolving images to match DNN classes produces a tremendous diversity of images. The mean DNN confidence scores for these images is 99.12% for the listed class, meaning that the DNN believes with near-certainty that the image is that type of thing. Shown are images selected to showcase diversity from 5 independent evolutionary runs. The images shed light on what the DNN network cares about, and what it does not, when classifying an image. For example, a school bus is alternating yellow and black lines, but does not need to have a windshield or wheels](https://i.sstatic.net/pBm48.png)](https://i.sstatic.net/pBm48.png)
  
  How this is possible? Can you please explain ideally in plain English?
  
  **Tags:** convolutional-neural-networks, computer-vision, image-recognition, deep-neural-networks, adversarial-ml
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105225/discussion-on-question-by-username_0-how-is-it-possible-that-deep-neural-networks-ar).  
  > **username_0**: Related video: ['How neural networks learn' - Part II: Adversarial Examples](https://www.youtube.com/watch?v=4rFOkpI0Lcg).  
  
  ---
  
  ## 9 answers
  
  ### Answer by username_12 on 2016-08-03. Score: 62 | Accepted answer
  
  First up, those images (even the first few) aren't complete trash despite being junk to humans; they're actually finely tuned with various advanced techniques, including another neural network.
  
  > The deep neural network is the pre-trained network modeled on AlexNet provided by [Caffe](https://github.com/BVLC/caffe). To evolve images, both the directly encoded and indirectly encoded images, we use the [Sferes](https://github.com/jbmouret/sferes2) evolutionary framework. The entire code base to conduct the evolutionary experiments can be download [sic] [here](https://github.com/Evolving-AI-Lab/fooling). The code for the images produced by gradient ascent is available [here](https://github.com/Evolving-AI-Lab/fooling/tree/master/caffe/ascent).
  
  Images that are actually random junk were correctly recognized as nothing meaningful:
  
  > In response to an unrecognizable image, the networks could have output a low confidence for each of the 1000 classes, instead of an extremely high confidence value for one of the classes. In fact, they do just that for randomly generated images (e.g. those in generation 0 of the evolutionary run)
  
  The original goal of the researchers was to use the neural networks to automatically generate images that look like the real things (by getting the recognizer's feedback and trying to change the image to get a more confident result), but they ended up creating the above art. Notice how even in the static-like images there are little splotches - usually near the center - which, it's fair to say, are triggering the recognition.
  
  > We were not trying to produce adversarial, unrecognizable images. Instead, we were trying to produce recognizable images, but these unrecognizable images emerged.
  
  Evidently, these images had just the right distinguishing features to match what the AI looked for in pictures. The "paddle" image does have a paddle-like shape, the "bagel" is round and the right color, the "projector" image is a camera-lens-like thing, the "computer keyboard" is a bunch of rectangles (like the individual keys), and the "chainlink fence" legitimately looks like a chain-link fence to me.
  
  > Figure 8. Evolving images to match DNN classes produces a tremendous diversity of images. Shown are images selected to showcase diversity from 5 evolutionary runs. The diversity suggests that the images are non-random, but that instead evolutions producing [sic] discriminative features of each target class.
  
  Further reading: [the original paper](http://www.evolvingai.org/files/DNNsEasilyFooled_cvpr15.pdf) (large PDF)
  
  > **Comments:**
  > **username_13**: The problem is - How DNN gives such high confidence ~ `99% !` that these some-object-pattern-like pictures are representing true objects ? Unless DNN is mentally ill and thinks it takes [Rorschach test](https://en.wikipedia.org/wiki/Rorschach_test) in some hospital :-)  
  > **username_13**: Sorry, I couldn't resist :-) "Bagel" is round, but so it is sun,eyeball,etc. "Projector" could be an atom model, laser beam cross section, etc. And so on and so on. So why the hell, bagel is preferred over sun, projector is preferred over atom schema, etc. with `99%` confidence ? For me - the main difference between DNN and human recognition is the fact that humans are not forced to recognize something, while NN seems are !  
  > **username_4**: They were not forced to put it in 1 of a 1000 categories. They gave 99% confidence of 1 category.  
  
  ---
  
  ### Answer by username_2 on 2016-08-06. Score: 29
  
  The images that you provided may be unrecognizable for us. They are actually the images that we recognize but evolved using the [Sferes](https://github.com/sferes2/sferes2) evolutionary framework.
  
  While these images are almost impossible for humans to label with anything but abstract arts, the Deep Neural Network will label them to be familiar objects with 99.99% confidence.
  
  This result highlights differences between how DNNs and humans recognize objects. Images are either directly (or indirectly) encoded
  
  According to this [video](https://youtu.be/M2IebCN9Ht4)
  
  > Changing an image originally correctly classified in a way imperceptible to humans can cause the cause DNN to classify it as something else.
  >
  > In the image below the number at the bottom are the images are supposed to look like the digits
  > But the network believes the images at the top (the one like username_4 noise) are real digits with 99.99% certainty.
  
  ![enter image description here](https://i.sstatic.net/Jx1wX.png)
  
  > The main reason why these are easily fooled is that Deep Neural Network does not see the world in the same way as human vision. We use the whole image to identify things while DNN depends on the features. As long as DNN detects certain features, it will classify the image as a familiar object it has been trained on.
  > The researchers proposed one way to prevent such fooling by adding the fooling images to the dataset in a new class and training DNN on the enlarged dataset. In the experiment, the confidence score decreases significantly for ImageNet AlexNet. It is not easy to fool the retrained DNN this time. But when the researchers applied such method to MNIST LeNet, evolution still produces many unrecognizable images with confidence scores of 99.99%.
  
  More details [here](http://www.evolvingai.org/fooling) and [here](http://www.kdnuggets.com/2015/01/deep-learning-can-be-easily-fooled.html).
  
  
  ---
  
  ### Answer by username_3 on 2017-09-18. Score: 17
  
  All answers here are great, but, for some reason, nothing has been said so far on why this effect *should not surprise* you. I'll fill the blank.
  
  Let me start with one requirement that is absolutely essential for this to work: the attacker *must know neural network architecture* (number of layers, size of each layer, etc). Moreover, in all cases that I examined myself, the attacker knows the snapshot of the model that is used in production, i.e. all weights. In other words, the "source code" of the network isn't a secret.
  
  You can't fool a neural network if you treat it like a black box. And you can't reuse the same fooling image for different networks.
  In fact, you have to "train" the target network yourself, and here by training I mean to run forward and backprop passes, but specially crafted for another purpose.
  
  ### Why is it working at all?
  
  Now, here's the intuition. Images are very high dimensional: even the space of small 32x32 color images has `3 * 32 * 32 = 3072` dimensions. But the training data set is relatively small and contains real pictures, all of which have some structure and nice statistical properties (e.g. smoothness of color). So the training data set is located on a tiny manifold of this huge space of images.
  
  The convolutional networks work extremely well on this manifold, but basically, know nothing about the rest of the space. The classification of the points outside of the manifold is just a linear extrapolation based on the points inside the manifold. No wonder that some particular points are extrapolated incorrectly. The attacker only needs a way to navigate to the closest of these points.
  
  ### Example
  
  Let me give you a concrete example how to fool a neural network. To make it compact, I'm going to use a very simple logistic regression network with one nonlinearity (sigmoid). It takes a 10-dimensional input `x`, computes a single number `p=sigmoid(W.dot(x))`, which is the probability of class 1 (versus class 0).
  
  ![logistic regression](https://i.sstatic.net/i3mO7.png)
  
  Suppose you know `W=(-1, -1, 1, -1, 1, -1, 1, 1, -1, 1)` and start with an input `x=(2, -1, 3, -2, 2, 2, 1, -4, 5, 1)`. A forward pass gives `sigmoid(W.dot(x))=0.0474` or 95% probability that `x` is class 0 example.
  
  ![x and W](https://i.sstatic.net/V6cNn.png)
  
  We'd like to find another example, `y`, which is very close to `x` but is classified by the network as 1. Note that `x` is 10-dimensional, so we have the freedom to nudge 10 values, which is a lot.
  
  Since `W[0]=-1` is negative, it's better for to have a small `y[0]` to make a total contribution of `y[0]*W[0]` small. Hence, let's make `y[0]=x[0]-0.5=1.5`.
  Likewise, `W[2]=1` is positive, so it's better to increase `y[2]` to make `y[2]*W[2]` bigger: `y[2]=x[2]+0.5=3.5`. And so on.
  
  ![x, W and y](https://i.sstatic.net/8zs12.png)
  
  The result is `y=(1.5, -1.5, 3.5, -2.5, 2.5, 1.5, 1.5, -3.5, 4.5, 1.5)`, and `sigmoid(W.dot(y))=0.88`. With this one change we improved the class 1 probability from 5% to 88%!
  
  ### Generalization
  
  If you look closely at the previous example, you'll notice that I knew exactly how to tweak `x` in order to move it to the target class, because I knew the network gradient. What I did was actually a *backpropagation*, but with respect to the data, instead of weights.
  
  In general, the attacker starts with target distribution `(0, 0, ..., 1, 0, ..., 0)` (zero everywhere, except for the class it wants to achieve), backpropagates to the data and makes a tiny move in that direction. Network state is not updated.
  
  Now it should be clear that it's a common feature of feed-forward networks that deal with a small data manifold, no matter how deep it is or the nature of data (image, audio, video or text).
  
  ### Potection
  
  The simplest way to prevent the system from being fooled is to use an ensemble of neural networks, i.e. a system that aggregates the votes of several networks on each request.
  It's much more difficult to backpropagate with respect to several networks simultaneously. The attacker might try to do it sequentially, one network at a time, but the update for one network might easily mess up
  with the results obtained for another network. The more networks are used, the more complex an attack becomes.
  
  Another possibility is to smooth the input before passing it to the network.
  
  ### Positive use of the same idea
  
  You shouldn't think that backpropagation to the image has only negative applications. A very similar technique, called *deconvolution*, is used for visualization and better understanding what neurons have learned.
  
  This technique allows synthesizing an image that causes a particular neuron to fire, basically see visually "what the neuron is looking for", which in general makes convolutional neural networks more interpretable.
  
  > **Comments:**
  > **username_4**: Reading the paper it seems that the recognizer was treated as a black box.  
  > **username_5**: If @username_4 is correct then this invalidates this otherwise great-looking answer. Do you have any comment on that?  
  
  ---
  
  ### Answer by username_6 on 2016-08-03. Score: 11
  
  An important question that does not yet have a satisfactory answer in neural network research is how DNNs come up with the predictions they offer. DNNs effectively work (though not exactly) by matching patches in the images to a "dictionary" of patches, one stored in each neuron (see [the youtube cat paper](http://research.google.com/archive/unsupervised_icml2012.html)). Thus, it may not have a high level view of the image since it only looks at patches, and images are usually downscaled to much lower resolution to obtain the results in current generation systems. Methods which look at how the components of the image interact may be able to avoid these problems.
  
  Some questions to ask for this work are: How confident were the networks when they made these predictions? How much volume do such adversarial images occupy in the space of all images?
  
  Some work I am aware of in this regard comes from Dhruv Batra and Devi Parikh's Lab at Virginia Tech who look into this for question answering systems: [Analyzing the Behavior of Visual Question Answering Models](http://arxiv.org/pdf/1606.07356v1.pdf) and [Interpreting Visual Question Answering models](https://filebox.ece.vt.edu/~dbatra/papers/gmpb_icmlvis16.pdf).
  
  More such work is needed, and just as the human visual system does also get fooled by such "optical illusions", these problems may be unavoidable if we use DNNs, though AFAIK nothing is yet known either way, theoretically or empirically.
  
  
  ---
  
  ### Answer by username_7 on 2018-09-06. Score: 5
  
  > How is it possible that deep neural networks are so easily fooled?
  >
  > Deep neural networks are easily fooled by giving high confidence predictions for unrecognizable images. How is this possible? Can you please explain ideally in **plain** English?
  
  Intuitively, extra hidden layers ought to make the network able to learn more complex classification functions, and thus do a better job classifying. While it may be named *deep learning* it's actually a shallow understanding.
  
  Test your own knowledge: Which animal in the grid below is a [Felis silvestris catus](https://en.wikipedia.org/wiki/Cat), take your time and no cheating. Here's a hint: which is a domestic house cat?
  
  [![Which is a domestic house cat?](https://i.sstatic.net/P09Tk.jpg)](https://i.sstatic.net/P09Tk.jpg)
  
  For a better understanding checkout: "[Adversarial Attack to Vulnerable Visualizations](https://medium.com/visweekly/adversarial-attack-to-vulnerable-visualizations-bfa3ed50efc2)" **and** "[Why are deep neural networks hard to train?](http://neuralnetworksanddeeplearning.com/chap5.html)".
  
  The problem is analogous to [aliasing](https://en.wikipedia.org/wiki/Aliasing), an effect that causes different signals to become indistinguishable (or aliases of one another) when sampled, and the [stagecoach-wheel effect](https://en.m.wikipedia.org/wiki/Wagon-wheel_effect), where a spoked wheel appears to rotate differently from its true rotation.
  
  The neural network doesn't *know* what it's looking at or which way it's going.
  
  Deep neural networks aren't an expert on something, they are trained to decide mathematically that some goal has been met, if they are not trained to reject wrong answers they don't have a concept of what is wrong; they only *know* what is correct and what is not correct - wrong and "not correct" are not necessarily the same thing, neither is "correct" and true.
  
  The neural network doesn't *know* right from wrong.
  
  Just like most people wouldn't know a house cat if they saw one, two or more, or none. How many house cats in the above photo grid, none. Any accusations of including cute cat pictures is unfounded, those are all dangerous wild animals.
  
  Here's another example. Does answering the question make Bart and Lisa smarter, does the person they are asking even know, are there unknown variables that can come into play?
  
  [![Are we there yet?](https://i.sstatic.net/BznDK.jpg)](https://i.sstatic.net/BznDK.jpg)
  
  We aren't there yet but neural networks can quickly provide an answer that is likely to be correct, especially if it was properly trained to avoid all misteps.
  
  
  ---
  
  ### Answer by username_8 on 2016-10-14. Score: 4
  
  Can't comment(due to that required 50 rep), but I wanted to make a response to username_2 and the OP. I think you guys are skipping the fact that the neural network only really is saying truly from a programmatic standpoint that "this is most like".
  
  For example, while we can list the above image examples as "abstract art", they definitively are most like was is listed. Remember learning algorithms have a scope on what they recognize as an object and if you look at all the above examples... and think about the scope of the algorithum... these make sense (even the ones at a glance we would recognize as username_4 noise). In username_2 example of the numbers, if you fuzz your eyes and bring the images out of focus, you can actually in every case spot patterns that really closely reflect the numbers in question.
  
  The problem that is being shown here is that the algorithm appears to not have a "unknown case". Basically when the pattern recognition says that it doesn't exist in the output scope. (so a final output node group that says this is nothing that I know off). For example, people do this as well, as it's one thing humans and learning algorithms have in common. Here's a link to show what I'm talking about (what is the following, define it) using only known animals that exist:
  
  ![Picture link](https://i.sstatic.net/AZwnpm.jpg)
  
  Now as a person, limited by what I know and can say, I'd have to conclude that the following is an elephant. But it's not. Learning algorithms (for the most part) do not have a "like a" statement, the out put always validates down to a confidence percentage. So tricking one in this fashion is not surprising... what is of course surprising is that based on it's knowledge set, it actually comes to the point in which, if you look at the above cases listed by OP and username_2 that a person... with a little looking... can see how the learning algorithm probable made the association.
  
  So, I wouldn't really call it a mislabel on the part of the algorithm, or even call it a case where it's been tricked... rather a case where it's scope was developed incorrectly.
  
  > **Comments:**
  > **username_4**: You are forgetting the 99% confidence level the researchers found.  
  > **username_9**: Indeed, an unknown image should be expected to return a low confidence level in every category. Is it a cat? No. Is it a dog? No. Is it a house? No. Is it a calculator? Sort of, it has all these buttons in a grid, but there's no screen and no writing on the buttons, so no.  
  
  ---
  
  ### Answer by username_0 on 2016-08-11. Score: 3
  
  The neural networks can be easily fooled or hacked by adding certain structured noise in image space ([Szegedy 2013](https://arxiv.org/abs/1312.6199), [Nguyen 2014](http://arxiv.org/abs/1412.1897)) due to ignoring non-discriminative information in their input.
  
  For example:
  
  > Learning to detect jaguars by matching the unique spots on their fur while ignoring the fact that they have four legs.[2015](http://arxiv.org/abs/1506.06579)
  
  So basically the high confidence prediction in certain models exists due to a '*combination of their locally linear nature and high-dimensional input space*'.[2015](http://arxiv.org/abs/1412.1897)
  
  Published as a conference paper at [ICLR 2015](http://www.stat.ucla.edu/~ywu/ICLR2015.pdf) (work by Dai) suggest that transferring discriminatively trained parameters to generative models, could be a great area for further improvements.
  
  
  ---
  
  ### Answer by username_10 on 2020-01-24. Score: 2
  
  Neural networks are easily fooled, provided you know how to fool them.
  
  Consider a linear network with an input layer and an output layer, which has an error function E (we don't need hidden layers to show how to fool a network). For a given input image x, E measures the (squared) difference between the network's output y and the desired (correct) output.
  
  The output unit’s state y is given by the inner product of x with the output unit’s weight vector w,so that
  
  ```
  y=w·x. 
  
  ```
  
  If we change x to x′ by adding ∆x then the output will change by ∆y to
  
  ```
  y′ = w·x+w·∆x (9.4) 
  
     = y + ∆y. (9.5)
  
  ```
  
  Notice that ∆x defines a direction in the input space; the question is, which direction ∆x will have most impact on y?
  
  By definition, a change in x in the direction ∇E produces the largest
  possible change in y. An adversarial image x' is constructed by taking the derivative ∇E of E with respect to the input image x, that is,
  
  ```
     x′ = x + ε∇E,
  
  ```
  
  where ε is a small constant.
  By definition, ∇E is the direction of steepest ascent, so the modification ε∇E to x will alter y more than a change in any other direction.
  
  [![enter image description here](https://i.sstatic.net/ggkFa.jpg)](https://i.sstatic.net/ggkFa.jpg)
  
  This is an extract from the book: [Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning](https://jim-username_10.staff.shef.ac.uk/BookBayes2012/books_by_jv_stone) (2019).
  
  
  ---
  
  ### Answer by username_11 on 2017-10-12. Score: 2
  
  There is already many good answers, I will just add to those that came before mine:
  
  This type of images you are referring to are called [adversarial perturbations](http://arxiv.org/pdf/1412.1897), (see [1](https://arxiv.org/pdf/1412.1897.pdf), and it is not limited to images, it has been shown to apply to text too, see [Jia & Liang, EMNLP 2017](https://arxiv.org/abs/1707.07328). In text, the introduction of an irrelevant sentence which doesn't contradict the paragraph has been seen to cause the network to come to a completely different answer (see see [Jia & Liang, EMNLP 2017](https://arxiv.org/abs/1707.07328)).
  
  **The reason they work** is due to the fact that neural network view images in a different way from us, coupled with the high dimentionality of the problem space. Where we see the whole picture, they see a combination of features which combine to form an object([Moosavi-Dezfooli et al., CVPR 2017](https://arxiv.org/abs/1610.08401)). According to perturbation generated against one network has been seen to have high likelihood to work on other networks:
  
  [![perturbations](https://i.sstatic.net/7zK85.jpg)](https://i.sstatic.net/7zK85.jpg)
  
  In the figure above, it is seen that The universal perturbations computed for the VGG-19 network, for example, have a fooling ratio above 53% for all other tested architectures.
  
  So how do you deal with the threat of adversarial perturbations?
  Well, for one, you can try to generate as many perturbations as you can and use them to fine-tune your model. Whist this somewhat solves the problem, it doesn't solve the problem entirely. In ([Moosavi-Dezfooli et al., CVPR 2017](https://arxiv.org/abs/1610.08401)) the author reported that, repeating the process by computing new perturbations and then fine-tuning again seems to yield no further improvements, regardless of the number of iterations, with the fooling ratio hovering around 80%.
  
  Perturbations are an indication of the shallow pattern matching that neural networks perform, coupled with their minimal lack of in-depth understanding of the problem at hand. More work still needs to be done.
  
  
  ---
  
  