I like the visualisations on this page: <a href="http:&#x2F;&#x2F;colah.github.io&#x2F;posts&#x2F;2014-03-NN-Manifolds-Topology&#x2F;" rel="nofollow">http:&#x2F;&#x2F;colah.github.io&#x2F;posts&#x2F;2014-03-NN-Manifolds-Topology&#x2F;</a> (Especially &quot;The Easy Way Out&quot; one.<p>It suggests that because neural networks cannot properly separate the classes, the often mush them together in the input-space so the &quot;panda&quot; region of image-space has loads of other classes speckled though it. If you purposely find those speckles you can make it think that a panda is a vulture.<p>That blog mentions adding smoothness constraints, so that you can&#x27;t suddenly go from panda to vulture with a tiny change in an image. But I wonder if an easier solution (ok, hack) is just to add different noise to the image 10 times or so, run it through the network and then combine the results.<p>Essentially you&#x27;d be doing Monte Carlo sampling of the image space around the input image. Or kind of blurring the image space.<p>Just an idea anyway. I don&#x27;t really know what I&#x27;m talking about.