Answers.Body: <p>It depends on what you mean by "know what is happening".</p>
  
  <p>Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as <span class="math-container">$1/1+e^{1/1+e^{\dots}}$</span>).</p>
  
  <p>However, if by 'know' you mean <em>predicting the output of some specific (black box) ANN</em>, by some other means, then the obstacle is the presence of chaos in a ANN that has <a href="http://sprott.physics.wisc.edu/pubs/paper234.pdf" rel="noreferrer">high degrees of freedom</a>.</p>
  
  <p>Here's also some relatively recent work by Hod Lipson on understanding ANNs through  <a href="http://arxiv.org/pdf/1506.06579.pdf" rel="noreferrer">visualisation</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2016-08-09T09:09:32.990Z
Answers.LastActivityDate: 2019-02-11T21:23:06.513Z
Answers.LastEditDate: 2019-02-11T21:23:06.513Z
Answers.UserDisplayName: NietzscheanAI
Answers.Body: <p>Short answer is <strong>no</strong>.</p>
  <p><em>Model interpretability</em> is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in  various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it...</p>
  <p>Here are some general (and recent, as of Dec 2017) resources on the subject:</p>
  <ul>
  <li><p>A recent (July 2017) article in Science provides a nice overview of the current status &amp; research: <a href="http://www.sciencemag.org/news/2017/07/how-ai-detectives-are-cracking-open-black-box-deep-learning" rel="nofollow noreferrer">How AI detectives are cracking open the black box of deep learning</a> (no in-text links, but googling names &amp; terms will pay off)</p>
  </li>
  <li><p>DARPA itself is currently running a program on <a href="https://www.darpa.mil/program/explainable-artificial-intelligence" rel="nofollow noreferrer">Explainable Artificial Intelligence (XAI)</a></p>
  </li>
  <li><p>There was a workshop in NIPS 2016 on <a href="http://nuit-blanche.blogspot.gr/2016/12/nips2016-interpretable-machine-learning.html" rel="nofollow noreferrer">Interpretable Machine Learning for Complex Systems</a>, as well as an <a href="http://people.csail.mit.edu/beenkim/icml_tutorial.html" rel="nofollow noreferrer">ICML 2017 tutorial on Interpretable Machine Learning</a> by <a href="https://beenkim.github.io/" rel="nofollow noreferrer">Been Kim</a> of Google Brain.</p>
  </li>
  </ul>
  <p>And on a more practical level (code etc):</p>
  <ul>
  <li><p>The What-If tool by Google, a brand new (September 2018) feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code (<a href="https://pair-code.github.io/what-if-tool/" rel="nofollow noreferrer">project page</a>, <a href="https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html" rel="nofollow noreferrer">blog post</a>)</p>
  </li>
  <li><p>The Layer-wise Relevance Propagation (LRP) toolbox for neural networks (<a href="http://www.jmlr.org/papers/v17/15-618.html" rel="nofollow noreferrer">paper</a>, <a href="http://www.explain-ai.org/" rel="nofollow noreferrer">project page</a>, <a href="https://github.com/sebastian-lapuschkin/lrp_toolbox" rel="nofollow noreferrer">code</a>, <a href="https://github.com/VigneshSrinivasan10/interprettensor" rel="nofollow noreferrer">TF Slim wrapper</a>)</p>
  </li>
  <li><p>FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs (<a href="http://blog.fastforwardlabs.com/2017/03/09/fairml-auditing-black-box-predictive-models.html" rel="nofollow noreferrer">blog post</a>, <a href="https://arxiv.org/abs/1611.04967" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/adebayoj/fairml" rel="nofollow noreferrer">code</a>)</p>
  </li>
  <li><p>LIME: Local Interpretable Model-agnostic Explanations (<a href="https://arxiv.org/abs/1602.04938" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/marcotcr/lime" rel="nofollow noreferrer">code</a>, <a href="https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime" rel="nofollow noreferrer">blog post</a>, <a href="https://cran.r-project.org/web/packages/lime/index.html" rel="nofollow noreferrer">R port</a>)</p>
  </li>
  <li><p>A very recent (November 2017) paper by Geoff Hinton, <a href="https://arxiv.org/abs/1711.09784" rel="nofollow noreferrer">Distilling a Neural Network Into a Soft Decision Tree</a>, with an independent <a href="https://github.com/kimhc6028/soft-decision-tree" rel="nofollow noreferrer">PyTorch implementation</a></p>
  </li>
  <li><p>SHAP: A Unified Approach to Interpreting Model Predictions (<a href="https://arxiv.org/abs/1705.07874" rel="nofollow noreferrer">paper</a>, authors' <a href="https://github.com/slundberg/shap" rel="nofollow noreferrer">Python code</a>, <a href="https://github.com/redichh/ShapleyR" rel="nofollow noreferrer">R package</a>)</p>
  </li>
  <li><p>Interpretable Convolutional Neural Networks (<a href="https://arxiv.org/abs/1710.00935" rel="nofollow noreferrer">paper</a>, authors' <a href="https://github.com/zqs1022/interpretableCNN" rel="nofollow noreferrer">code</a>)</p>
  </li>
  <li><p>Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google (<a href="https://github.com/tensorflow/lucid" rel="nofollow noreferrer">code</a>; papers: <a href="https://distill.pub/2017/feature-visualization/" rel="nofollow noreferrer">Feature Visualization</a>, <a href="https://distill.pub/2018/building-blocks/" rel="nofollow noreferrer">The Building Blocks of Interpretability</a>)</p>
  </li>
  <li><p>Transparecy-by-Design (TbD) networks (<a href="https://arxiv.org/abs/1803.05268" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/davidmascharka/tbd-nets" rel="nofollow noreferrer">code</a>, <a href="https://hub.mybinder.org/user/davidmascharka-tbd-nets-nex18rln/notebooks/full-vqa-example.ipynb" rel="nofollow noreferrer">demo</a>)</p>
  </li>
  <li><p>SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability (<a href="https://arxiv.org/abs/1706.05806" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/google/svcca" rel="nofollow noreferrer">code</a>, <a href="https://ai.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html" rel="nofollow noreferrer">Google blog post</a>)</p>
  </li>
  <li><p>TCAV: Testing with Concept Activation Vectors (<a href="https://arxiv.org/abs/1711.11279" rel="nofollow noreferrer">ICML 2018 paper</a>, <a href="https://github.com/tensorflow/tcav/" rel="nofollow noreferrer">Tensorflow code</a>)</p>
  </li>
  <li><p>Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization (<a href="https://arxiv.org/abs/1610.02391" rel="nofollow noreferrer">paper</a>, authors' <a href="https://github.com/ramprs/grad-cam" rel="nofollow noreferrer">Torch code</a>, <a href="https://github.com/Ankush96/grad-cam.tensorflow" rel="nofollow noreferrer">Tensorflow code</a>, <a href="https://github.com/meliketoy/gradcam.pytorch" rel="nofollow noreferrer">PyTorch code</a>, Keras <a href="http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb" rel="nofollow noreferrer">example notebook</a>)</p>
  </li>
  <li><p>Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL (<a href="http://netdissect.csail.mit.edu/" rel="nofollow noreferrer">project page</a>, <a href="https://github.com/CSAILVision/NetDissect" rel="nofollow noreferrer">Caffe code</a>, <a href="https://github.com/CSAILVision/NetDissect-Lite" rel="nofollow noreferrer">PyTorch port</a>)</p>
  </li>
  <li><p>GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL (<a href="https://gandissect.csail.mit.edu/" rel="nofollow noreferrer">project page</a>, with links to paper &amp; code)</p>
  </li>
  <li><p>Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions (<a href="https://arxiv.org/abs/1811.08011" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/gudovskiy/e2x" rel="nofollow noreferrer">code</a>)</p>
  </li>
  <li><p>InterpretML by Microsoft (<a href="https://github.com/interpretml/interpret" rel="nofollow noreferrer">code</a> still in alpha)</p>
  </li>
  <li><p>Anchors: High-Precision Model-Agnostic Explanations (<a href="https://arxiv.org/abs/1905.07697" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/marcotcr/anchor" rel="nofollow noreferrer">code</a>)</p>
  </li>
  <li><p>Diverse Counterfactual Explanations (DiCE) by Microsoft (<a href="https://arxiv.org/abs/1905.07697" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/interpretml/DiCE" rel="nofollow noreferrer">code</a>, <a href="https://www.microsoft.com/en-us/research/blog/open-source-library-provides-explanation-for-machine-learning-through-diverse-counterfactuals/" rel="nofollow noreferrer">blog post</a>)</p>
  </li>
  <li><p>Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method (<a href="https://arxiv.org/abs/2008.02312" rel="nofollow noreferrer">paper</a>, <a href="https://github.com/Fu0511/XGrad-CAM" rel="nofollow noreferrer">code</a>)</p>
  </li>
  </ul>
  <p>Lately, there has been a surge of interest to start building a more theoretical basis for deep learning neural nets. In this context, renowned statistician and compressive sensing pioneer David Donoho has very recently (fall 2017) started offering a course at Stanford, <a href="https://stats385.github.io/" rel="nofollow noreferrer">Theories of Deep Learning (STATS 385)</a>, with almost all the material available online; it is highly recommended...</p>
  <p><strong>UPDATES</strong>:</p>
  <ul>
  <li><a href="https://christophm.github.io/interpretable-ml-book/" rel="nofollow noreferrer">Interpretable Machine Learning</a>, an online Gitbook by Christoph Molnar with <a href="https://github.com/christophM/iml" rel="nofollow noreferrer">R code</a> (although in mostly covers algorithms other than neural networks)</li>
  <li>A <a href="https://twitter.com/ledell/status/995930308947140608" rel="nofollow noreferrer">Twitter thread</a>, linking to several interpretation tools available for R.</li>
  <li>A short (4 hrs) online course by Kaggle, <a href="https://www.kaggle.com/learn/machine-learning-explainability" rel="nofollow noreferrer">Machine Learning Explainability</a>, and the accompanying <a href="https://towardsdatascience.com/why-model-explainability-is-the-next-data-science-superpower-b11b6102a5e0" rel="nofollow noreferrer">blog post</a></li>
  <li>A new ICML 2018 Tutorial, <a href="http://unsupervised.cs.princeton.edu/deeplearningtutorial.html" rel="nofollow noreferrer">Toward theoretical understanding of deep learning</a>, by Sanjeev Arora</li>
  <li>A <strong>whole bunch</strong> of resources in the <a href="https://github.com/jphall663/awesome-machine-learning-interpretability" rel="nofollow noreferrer">Awesome Machine Learning Interpretability repo</a></li>
  </ul>
  <p><strong>NOTE</strong>: I do no longer keep this answer updated; for updates, see my answer in <a href="https://ai.stackexchange.com/questions/12870/which-explainable-artificial-intelligence-techniques-are-there/24138#24138">Which explainable artificial intelligence techniques are there?</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T14:08:21.467Z
Answers.Text: Hi. This seems to be a good answer, but you need to clean it up and organise a little bit. The first resources should be the most helpful and general. Then you can list more specific resources and research papers, IMHO. And later you can list e.g. Twitter threads or whatever.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-12-12T16:41:43.700Z
Answers.LastActivityDate: 2021-04-16T10:08:19.733Z
Answers.LastEditDate: 2021-04-16T10:08:19.733Z
Answers.UserDisplayName: desertnaut
Answers.Body: <p>I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks.  That is, we understand something of the <em>how</em> they work (for example, the math behind back propagation) but we don't really understand <em>why</em> they work.  It's sort of a subtle distinction, but the point is that no, we don't understand the very deepest details of how exactly you go from a bunch of weights, to, say, recognizing a cat playing with a ball.  </p>
  
  <p>At least in terms of image recognition, the best explanation I've heard is that successive layers of a neural network learn more sophisticated features, composed of the more granular features from earlier levels.  That is to say, the first layer might recognize "edges" or "straight lines".  The next layer might then learn geometric shapes like "box", or "triangle", and then a higher layer might learn "nose" or "eye" based on those earlier features, and then a higher level layer still learns "face" made up from "eye", "nose", "jaw", etc.   But even that, as I understand it, is still hypothetical and/or not understood in complete detail. </p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-10T16:09:33.697Z
Answers.Text: I'd be interested to read the actual quotes. At the broadest conceptual level, the *why* is "They are Universal function approximators trained to reduce the error in a regression problem".
Answers.UserDisplayName: NietzscheanAI
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-10T17:31:47.513Z
Answers.Text: I'll see if I can track them down.  I'm pretty sure the quote from Geoffrey Hinton that I'm thinking of is in a video.. either from his Coursera Class or some video he has up on Youtube.  If I can find it, I'll edit my answer and link it in.
Answers.UserDisplayName: mindcrime
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-16T16:28:40.817Z
Answers.Text: I haven't forgotten.  I'll try to find them when I have a little free time.   I think at least one of the ones I'm thinking of was from a video that's part of a Coursera course.
Answers.UserDisplayName: mindcrime
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-17T17:34:47.383Z
Answers.Text: This [study](https://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf) can help to put same references: 'However there is no clear understanding of
  why they perform so well, or how they might be improved'.
Answers.UserDisplayName: kenorb
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-10T16:03:27.817Z
Answers.LastActivityDate: 2016-08-10T16:03:27.817Z
Answers.UserDisplayName: mindcrime
Answers.Body: <p>Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise. </p>
  
  <p>See <a href="https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html" rel="nofollow noreferrer">Inceptionism: Going Deeper into Neural Networks (Google Research Blog)</a>.</p>
  
  <p>This kind of represents what the network knows.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-09T10:28:31.650Z
Answers.LastActivityDate: 2018-03-14T21:24:06.093Z
Answers.LastEditDate: 2018-03-14T21:24:06.093Z
Answers.UserDisplayName: DiCaprio
Answers.Body: <p>Here is an answer by Carlos E. Perez to the question <a href="https://www.quora.com/What-is-theory-behind-deep-learning/answer/Carlos-E-Perez?srid=CpS&amp;share=0f965f46" rel="noreferrer">What is theory behind deep learning?</a></p>
  
  <blockquote>
    <p>[...]</p>
    
    <p>The underlying mathematics of Deep Learning has been in existence for several decades, however the impressive results that we see today are part a consequence of much faster hardware, more data and incremental improvements in methods. </p>
    
    <p>Deep Learning in general can be framed as optimization problem where the objective is a function of the model error. This optimization problem is very difficult to solve consider that the parameter space of the model (i.e. weights of the neural network) leads to a problem in extremely high dimension. An optimization algorithm could take a very long time to explore this space. Furthermore, there was an unverified belief that the problem was non-convex and computation would forever be stuck in local minima.</p>
    
    <p>[...]</p>
    
    <p>The theory of why machines actually converge to an attractor or in other words learn to recognize complex patterns is still unknown.</p>
  </blockquote>
  
  <p>To sum up: we have some ideas, but we're not quite sure.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-08-26T10:34:16.803Z
Answers.LastActivityDate: 2016-08-26T10:34:16.803Z
Answers.UserDisplayName: Benjamin Crouzier
Answers.Body: <blockquote>
    <p>Do scientists know what is happening inside artificial neural networks?</p>
  </blockquote>
  
  <h3>YES</h3>
  
  <blockquote>
    <p>Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant?</p>
  </blockquote>
  
  <p>I guess "to know from the kitchen" means "to know in detail"?</p>
  
  <p>Let me give you a series of analogies:</p>
  
  <ol>
  <li>Does an airplane engineer know from the kitchen what happens inside the airplane?</li>
  <li>Does a chip designer know in detail what happens in the chip (s)he designed?</li>
  <li>Does a civil engineer know everything about the house he constructed?</li>
  </ol>
  
  <p>The devil is in the detail, but a crucial point here is that it's about artificial structures. They don't randomly appear. You need a lot of knowledge to get anything useful. For Neural Networks, I would say it took roughly 40 years from the publication of the key idea (Rosenblatt perceptron, 1957) to the first application (US Postal Service, 1989). And from there again 13 years of active reserach to really impressive systems (ImageNet 2012).</p>
  
  <p><strong>What we know super well is how the training works</strong>. Because it needs to be implemented. So on a very small structure, we know it in detail.</p>
  
  <p>Think of computers. The chip designers know very well how their chip works. But they will likely only have a very rough idea how the Linux operating system works.</p>
  
  <p>Another example is physics and chemistry: Physics describes the core forces of the universe. Does that mean they know everything about chemistry as well? Hell no! A "perfect" physicist can explain everything in chemistry ... but it would be pretty much useless. He would need a lot more information, not be able to skip the irrelevant parts. Simply because he "zoomed in" too much - considers details which are in practice neither interesting nor important. Please note that the knowledge of the physicist is not wrong. Maybe one could even deduce the knowledge from the chemist from it. But this "high-level" understanding of molecule interaction is missing.</p>
  
  <p>The key insight from those two examples are abstraction layers: <strong>You can build complexity from simple structures</strong>.</p>
  
  <p>What else?</p>
  
  <p><strong>We know well what is in principle achievable with the neural networks</strong> we design:</p>
  
  <ul>
  <li>A neural network designed to play Go - no matter how sophisticated - will never even be able to play chess. You can, of course, add another abstraction layer around it and combine things. But this approach needs humans.</li>
  <li>A neural network designed for distinguishing dogs from cats which has only seen pudels and Persian cats will likely perform really bad when it has to decide for Yorkshire Terriers.</li>
  </ul>
  
  <p>Oh, and of course we have analytical approaches for neural networks. I wrote my masters thesis about <a href="https://arxiv.org/pdf/1707.09725.pdf" rel="nofollow noreferrer">Analysis and Optimization of Convolutional Neural Network Architectures</a>. In this context <a href="https://github.com/marcotcr/lime" rel="nofollow noreferrer">LIME</a> (Local Interpretable Model-Agnostic Explanations) is nice:</p>
  
  <p><a href="https://i.sstatic.net/wXzfn.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wXzfn.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-25T18:07:35.910Z
Answers.Text: Think of early day automobile / aircraft engineers. Would you say they don't know what's happening inside their aircraft / automobile, because they didn't built them, because their shape was not aerodynamic?
Answers.UserDisplayName: Martin Thoma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-09-25T16:56:41.647Z
Answers.LastActivityDate: 2018-09-25T16:56:41.647Z
Answers.UserDisplayName: Martin Thoma
Answers.Body: <p>There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically <em>convolutional neural networks</em> that you've mentioned.</p>
  
  <h3>Visualizing the activations and layer weights</h3>
  
  <p><a href="https://medium.com/@awjuliani/visualizing-neural-network-layer-activation-tensorflow-tutorial-d45f8bf7bbc4" rel="noreferrer">Activations visualization</a> is the first obvious and straight-forward one. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse (most values are zero) and localized. This sometimes shows what exactly a particular layer is focused on when it sees an image.</p>
  
  <p>Another great work on activations that I'd like to mention is <a href="https://www.youtube.com/watch?v=AgkfIQ4IGaM" rel="noreferrer">deepvis</a> that shows reaction of every neuron at each layer, including pooling and normalization layers. Here's how they <a href="http://yosinski.com/deepvis" rel="noreferrer">describe it</a>:</p>
  
  <blockquote>
    <p>In short, we’ve gathered a few different methods that allow you to
    “triangulate” what feature a neuron has learned, which can help you
    better understand how DNNs work.</p>
  </blockquote>
  
  <p>The second common strategy is to visualize the weights (filters). These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. For example, the first layer usually learns gabor-like filters that basically detect edges and blobs.</p>
  
  <p><a href="https://i.sstatic.net/D6pXk.jpg" rel="noreferrer"><img src="https://i.sstatic.net/D6pXk.jpg" alt="first layer filters"></a></p>
  
  <h3>Occlusion experiments</h3>
  
  <p>Here's the idea. Suppose that a ConvNet classifies an image as a dog. How can we be certain that it’s actually picking up on the dog in the image as opposed to some contextual cues from the background or some other miscellaneous object? </p>
  
  <p>One way of investigating which part of the image some classification prediction is coming from is by plotting the probability of the class of interest (e.g. dog class) as a function of the position of an occluder object. 
  If we iterate over regions of the image, replace it with all zeros and check the classification result, we can build a 2-dimensional heat map of what's most important for the network on a particular image. This approach has been used in <a href="https://arxiv.org/pdf/1311.2901.pdf" rel="noreferrer">Matthew Zeiler’s Visualizing and Understanding Convolutional Networks</a> (that you refer to in your question):</p>
  
  <p><a href="https://i.sstatic.net/O10WA.jpg" rel="noreferrer"><img src="https://i.sstatic.net/O10WA.jpg" alt="occlusion experiments"></a></p>
  
  <h3>Deconvolution</h3>
  
  <p>Another approach is to synthesize an image that causes a particular neuron to fire, basically what the neuron is looking for. The idea is to compute the gradient with respect to the image, instead of the usual gradient with respect to the weights. So you pick a layer, set the gradient there to be all zero except for one for one neuron and backprop to the image. </p>
  
  <p>Deconv actually does something called <em>guided backpropagation</em> to make a nicer looking image, but it's just a detail.</p>
  
  <h3>Similar approaches to other neural networks</h3>
  
  <p>Highly recommend <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" rel="noreferrer">this post by Andrej Karpathy</a>, in which he plays a lot with Recurrent Neural Networks (RNN). In the end, he applies a similar technique to see what the neurons actually learn:</p>
  
  <blockquote>
    <p>The neuron highlighted in this image seems to get very excited about
    URLs and turns off outside of the URLs. The LSTM is likely using this
    neuron to remember if it is inside a URL or not.</p>
  </blockquote>
  
  <h3>Conclusion</h3>
  
  <p>I've mentioned only a small fraction of results in this area of research. It's pretty active and new methods that shed light to the neural network inner workings appear each year.</p>
  
  <p>To answer your question, there's always something that scientists don't know yet, but in many cases they have a good picture (literary) of what's going on inside and can answer many particular questions. </p>
  
  <p>To me the quote from your question simply highlights the importance of research of not only accuracy improvement, but the inner structure of the network as well. As Matt Zieler tells in <a href="https://www.youtube.com/watch?v=ghEmQSxT6tw" rel="noreferrer">this talk</a>, sometimes a good visualization can lead, in turn, to better accuracy.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-17T10:03:18.140Z
Answers.LastActivityDate: 2017-09-17T10:03:18.140Z
Answers.UserDisplayName: Maxim
Body: <p>Do scientists or research experts know from the kitchen what is happening inside complex &quot;deep&quot; neural network with at least millions of connections firing at an instant? Do they understand the process behind this (e.g. what is happening inside and how it works exactly), or it is a subject of debate?</p>
  <p>For example this <a href="https://www.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf" rel="noreferrer">study</a> says:</p>
  <blockquote>
  <p>However there is no clear understanding of <em>why</em> they perform so well, or <em>how</em> they might be improved.</p>
  </blockquote>
  <p>So does this mean that scientists actually don't know how complex convolutional network models work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-31T12:02:25.697Z
Comments.Text: "*why they perform so well*" - they don't really perform SO well. As with most new technology, failures are underreported.
Comments.UserDisplayName: Tomáš Zato
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-15T15:01:53.663Z
Comments.Text: Take a look at [this question](https://ai.stackexchange.com/q/12870/2444) too.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-22T20:51:16.993Z
Comments.Text: Video: ['How neural networks learn' - Part I: Feature Visualization](https://www.youtube.com/watch?v=McgxRxi2Jqo).
Comments.UserDisplayName: kenorb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-10-12T13:03:34.643Z
Comments.Text: An article of mine on this question published in Queen's Quarterly (but full text on my website...): [The Curtain](https://unfamiliarplace.com/wp-content/uploads/2022/04/The-Curtain.pdf)
Comments.UserDisplayName: Luke Sawczak
Tags: <neural-networks><deep-learning><convolutional-neural-networks><explainable-ai>
Title: Do scientists know what is happening inside artificial neural networks?
UserDisplayName: kenorb
ThreadText: # Do scientists know what is happening inside artificial neural networks?
  
  (Asked by: username_0 on 2016-08-09)
  
  Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant? Do they understand the process behind this (e.g. what is happening inside and how it works exactly), or it is a subject of debate?
  
  For example this [study](https://www.cs.nyu.edu/%7Efergus/papers/zeilerECCV2014.pdf) says:
  
  > However there is no clear understanding of *why* they perform so well, or *how* they might be improved.
  
  So does this mean that scientists actually don't know how complex convolutional network models work?
  
  **Tags:** neural-networks, deep-learning, convolutional-neural-networks, explainable-ai
  
  > **Comments:**
  > **username_1**: "*why they perform so well*" - they don't really perform SO well. As with most new technology, failures are underreported.  
  > **username_2**: Take a look at [this question](https://ai.stackexchange.com/q/12870/2444) too.  
  > **username_0**: Video: ['How neural networks learn' - Part I: Feature Visualization](https://www.youtube.com/watch?v=McgxRxi2Jqo).  
  > **username_3**: An article of mine on this question published in Queen's Quarterly (but full text on my website...): [The Curtain](https://unfamiliarplace.com/wp-content/uploads/2022/04/The-Curtain.pdf)  
  
  ---
  
  ## 7 answers
  
  ### Answer by username_10 on 2017-09-17. Score: 73 | Accepted answer
  
  There are many approaches that aim to make a trained neural network more interpretable and less like a "black box", specifically *convolutional neural networks* that you've mentioned.
  
  ### Visualizing the activations and layer weights
  
  [Activations visualization](https://medium.com/@awjuliani/visualizing-neural-network-layer-activation-tensorflow-tutorial-d45f8bf7bbc4) is the first obvious and straight-forward one. For ReLU networks, the activations usually start out looking relatively blobby and dense, but as the training progresses the activations usually become more sparse (most values are zero) and localized. This sometimes shows what exactly a particular layer is focused on when it sees an image.
  
  Another great work on activations that I'd like to mention is [deepvis](https://www.youtube.com/watch?v=AgkfIQ4IGaM) that shows reaction of every neuron at each layer, including pooling and normalization layers. Here's how they [describe it](http://yosinski.com/deepvis):
  
  > In short, we’ve gathered a few different methods that allow you to
  > “triangulate” what feature a neuron has learned, which can help you
  > better understand how DNNs work.
  
  The second common strategy is to visualize the weights (filters). These are usually most interpretable on the first CONV layer which is looking directly at the raw pixel data, but it is possible to also show the filter weights deeper in the network. For example, the first layer usually learns gabor-like filters that basically detect edges and blobs.
  
  [![first layer filters](https://i.sstatic.net/D6pXk.jpg)](https://i.sstatic.net/D6pXk.jpg)
  
  ### Occlusion experiments
  
  Here's the idea. Suppose that a ConvNet classifies an image as a dog. How can we be certain that it’s actually picking up on the dog in the image as opposed to some contextual cues from the background or some other miscellaneous object?
  
  One way of investigating which part of the image some classification prediction is coming from is by plotting the probability of the class of interest (e.g. dog class) as a function of the position of an occluder object.
  If we iterate over regions of the image, replace it with all zeros and check the classification result, we can build a 2-dimensional heat map of what's most important for the network on a particular image. This approach has been used in [Matthew Zeiler’s Visualizing and Understanding Convolutional Networks](https://arxiv.org/pdf/1311.2901.pdf) (that you refer to in your question):
  
  [![occlusion experiments](https://i.sstatic.net/O10WA.jpg)](https://i.sstatic.net/O10WA.jpg)
  
  ### Deconvolution
  
  Another approach is to synthesize an image that causes a particular neuron to fire, basically what the neuron is looking for. The idea is to compute the gradient with respect to the image, instead of the usual gradient with respect to the weights. So you pick a layer, set the gradient there to be all zero except for one for one neuron and backprop to the image.
  
  Deconv actually does something called *guided backpropagation* to make a nicer looking image, but it's just a detail.
  
  ### Similar approaches to other neural networks
  
  Highly recommend [this post by Andrej Karpathy](http://karpathy.github.io/2015/05/21/rnn-effectiveness/), in which he plays a lot with Recurrent Neural Networks (RNN). In the end, he applies a similar technique to see what the neurons actually learn:
  
  > The neuron highlighted in this image seems to get very excited about
  > URLs and turns off outside of the URLs. The LSTM is likely using this
  > neuron to remember if it is inside a URL or not.
  
  ### Conclusion
  
  I've mentioned only a small fraction of results in this area of research. It's pretty active and new methods that shed light to the neural network inner workings appear each year.
  
  To answer your question, there's always something that scientists don't know yet, but in many cases they have a good picture (literary) of what's going on inside and can answer many particular questions.
  
  To me the quote from your question simply highlights the importance of research of not only accuracy improvement, but the inner structure of the network as well. As Matt Zieler tells in [this talk](https://www.youtube.com/watch?v=ghEmQSxT6tw), sometimes a good visualization can lead, in turn, to better accuracy.
  
  
  ---
  
  ### Answer by username_4 on 2016-08-09. Score: 32
  
  It depends on what you mean by "know what is happening".
  
  Conceptually, yes: ANN perform nonlinear regression. The actual expression represented by the weight matrix/activation function(s) of an ANN can be explicitly expanded in symbolic form (e.g. containing sub-expressions such as $1/1+e^{1/1+e^{\dots}}$).
  
  However, if by 'know' you mean *predicting the output of some specific (black box) ANN*, by some other means, then the obstacle is the presence of chaos in a ANN that has [high degrees of freedom](http://sprott.physics.wisc.edu/pubs/paper234.pdf).
  
  Here's also some relatively recent work by Hod Lipson on understanding ANNs through [visualisation](http://arxiv.org/pdf/1506.06579.pdf).
  
  
  ---
  
  ### Answer by username_5 on 2017-12-12. Score: 21
  
  Short answer is **no**.
  
  *Model interpretability* is a hyper-active and hyper-hot area of current research (think of holy grail, or something), which has been brought forward lately not least due to the (often tremendous) success of deep learning models in various tasks; these models are currently only black boxes, and we naturally feel uncomfortable about it...
  
  Here are some general (and recent, as of Dec 2017) resources on the subject:
  
  * A recent (July 2017) article in Science provides a nice overview of the current status & research: [How AI detectives are cracking open the black box of deep learning](http://www.sciencemag.org/news/2017/07/how-ai-detectives-are-cracking-open-black-box-deep-learning) (no in-text links, but googling names & terms will pay off)
  * DARPA itself is currently running a program on [Explainable Artificial Intelligence (XAI)](https://www.darpa.mil/program/explainable-artificial-intelligence)
  * There was a workshop in NIPS 2016 on [Interpretable Machine Learning for Complex Systems](http://nuit-blanche.blogspot.gr/2016/12/nips2016-interpretable-machine-learning.html), as well as an [ICML 2017 tutorial on Interpretable Machine Learning](http://people.csail.mit.edu/beenkim/icml_tutorial.html) by [Been Kim](https://beenkim.github.io/) of Google Brain.
  
  And on a more practical level (code etc):
  
  * The What-If tool by Google, a brand new (September 2018) feature of the open-source TensorBoard web application, which let users analyze an ML model without writing code ([project page](https://pair-code.github.io/what-if-tool/), [blog post](https://ai.googleblog.com/2018/09/the-what-if-tool-code-free-probing-of.html))
  * The Layer-wise Relevance Propagation (LRP) toolbox for neural networks ([paper](http://www.jmlr.org/papers/v17/15-618.html), [project page](http://www.explain-ai.org/), [code](https://github.com/sebastian-lapuschkin/lrp_toolbox), [TF Slim wrapper](https://github.com/VigneshSrinivasan10/interprettensor))
  * FairML: Auditing Black-Box Predictive Models, by Cloudera Fast Forward Labs ([blog post](http://blog.fastforwardlabs.com/2017/03/09/fairml-auditing-black-box-predictive-models.html), [paper](https://arxiv.org/abs/1611.04967), [code](https://github.com/adebayoj/fairml))
  * LIME: Local Interpretable Model-agnostic Explanations ([paper](https://arxiv.org/abs/1602.04938), [code](https://github.com/marcotcr/lime), [blog post](https://www.oreilly.com/learning/introduction-to-local-interpretable-model-agnostic-explanations-lime), [R port](https://cran.r-project.org/web/packages/lime/index.html))
  * A very recent (November 2017) paper by Geoff Hinton, [Distilling a Neural Network Into a Soft Decision Tree](https://arxiv.org/abs/1711.09784), with an independent [PyTorch implementation](https://github.com/kimhc6028/soft-decision-tree)
  * SHAP: A Unified Approach to Interpreting Model Predictions ([paper](https://arxiv.org/abs/1705.07874), authors' [Python code](https://github.com/slundberg/shap), [R package](https://github.com/redichh/ShapleyR))
  * Interpretable Convolutional Neural Networks ([paper](https://arxiv.org/abs/1710.00935), authors' [code](https://github.com/zqs1022/interpretableCNN))
  * Lucid, a collection of infrastructure and tools for research in neural network interpretability by Google ([code](https://github.com/tensorflow/lucid); papers: [Feature Visualization](https://distill.pub/2017/feature-visualization/), [The Building Blocks of Interpretability](https://distill.pub/2018/building-blocks/))
  * Transparecy-by-Design (TbD) networks ([paper](https://arxiv.org/abs/1803.05268), [code](https://github.com/davidmascharka/tbd-nets), [demo](https://hub.mybinder.org/user/davidmascharka-tbd-nets-nex18rln/notebooks/full-vqa-example.ipynb))
  * SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability ([paper](https://arxiv.org/abs/1706.05806), [code](https://github.com/google/svcca), [Google blog post](https://ai.googleblog.com/2017/11/interpreting-deep-neural-networks-with.html))
  * TCAV: Testing with Concept Activation Vectors ([ICML 2018 paper](https://arxiv.org/abs/1711.11279), [Tensorflow code](https://github.com/tensorflow/tcav/))
  * Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization ([paper](https://arxiv.org/abs/1610.02391), authors' [Torch code](https://github.com/ramprs/grad-cam), [Tensorflow code](https://github.com/Ankush96/grad-cam.tensorflow), [PyTorch code](https://github.com/meliketoy/gradcam.pytorch), Keras [example notebook](http://nbviewer.jupyter.org/github/fchollet/deep-learning-with-python-notebooks/blob/master/5.4-visualizing-what-convnets-learn.ipynb))
  * Network Dissection: Quantifying Interpretability of Deep Visual Representations, by MIT CSAIL ([project page](http://netdissect.csail.mit.edu/), [Caffe code](https://github.com/CSAILVision/NetDissect), [PyTorch port](https://github.com/CSAILVision/NetDissect-Lite))
  * GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, by MIT CSAIL ([project page](https://gandissect.csail.mit.edu/), with links to paper & code)
  * Explain to Fix: A Framework to Interpret and Correct DNN Object Detector Predictions ([paper](https://arxiv.org/abs/1811.08011), [code](https://github.com/gudovskiy/e2x))
  * InterpretML by Microsoft ([code](https://github.com/interpretml/interpret) still in alpha)
  * Anchors: High-Precision Model-Agnostic Explanations ([paper](https://arxiv.org/abs/1905.07697), [code](https://github.com/marcotcr/anchor))
  * Diverse Counterfactual Explanations (DiCE) by Microsoft ([paper](https://arxiv.org/abs/1905.07697), [code](https://github.com/interpretml/DiCE), [blog post](https://www.microsoft.com/en-us/research/blog/open-source-library-provides-explanation-for-machine-learning-through-diverse-counterfactuals/))
  * Axiom-based Grad-CAM (XGrad-CAM): Towards Accurate Visualization and Explanation of CNNs, a refinement of the existing Grad-CAM method ([paper](https://arxiv.org/abs/2008.02312), [code](https://github.com/Fu0511/XGrad-CAM))
  
  Lately, there has been a surge of interest to start building a more theoretical basis for deep learning neural nets. In this context, renowned statistician and compressive sensing pioneer David Donoho has very recently (fall 2017) started offering a course at Stanford, [Theories of Deep Learning (STATS 385)](https://stats385.github.io/), with almost all the material available online; it is highly recommended...
  
  **UPDATES**:
  
  * [Interpretable Machine Learning](https://christophm.github.io/interpretable-ml-book/), an online Gitbook by Christoph Molnar with [R code](https://github.com/christophM/iml) (although in mostly covers algorithms other than neural networks)
  * A [Twitter thread](https://twitter.com/ledell/status/995930308947140608), linking to several interpretation tools available for R.
  * A short (4 hrs) online course by Kaggle, [Machine Learning Explainability](https://www.kaggle.com/learn/machine-learning-explainability), and the accompanying [blog post](https://towardsdatascience.com/why-model-explainability-is-the-next-data-science-superpower-b11b6102a5e0)
  * A new ICML 2018 Tutorial, [Toward theoretical understanding of deep learning](http://unsupervised.cs.princeton.edu/deeplearningtutorial.html), by Sanjeev Arora
  * A **whole bunch** of resources in the [Awesome Machine Learning Interpretability repo](https://github.com/jphall663/awesome-machine-learning-interpretability)
  
  **NOTE**: I do no longer keep this answer updated; for updates, see my answer in [Which explainable artificial intelligence techniques are there?](https://ai.stackexchange.com/questions/12870/which-explainable-artificial-intelligence-techniques-are-there/24138#24138)
  
  > **Comments:**
  > **username_2**: Hi. This seems to be a good answer, but you need to clean it up and organise a little bit. The first resources should be the most helpful and general. Then you can list more specific resources and research papers, IMHO. And later you can list e.g. Twitter threads or whatever.  
  
  ---
  
  ### Answer by username_6 on 2016-08-10. Score: 12
  
  I'm afraid I don't have the specific citations handy, but I have seen/heard quotes by experts like Andrew Ng and Geoffrey Hinton where they clearly say that we do not really understand neural networks. That is, we understand something of the *how* they work (for example, the math behind back propagation) but we don't really understand *why* they work. It's sort of a subtle distinction, but the point is that no, we don't understand the very deepest details of how exactly you go from a bunch of weights, to, say, recognizing a cat playing with a ball.
  
  At least in terms of image recognition, the best explanation I've heard is that successive layers of a neural network learn more sophisticated features, composed of the more granular features from earlier levels. That is to say, the first layer might recognize "edges" or "straight lines". The next layer might then learn geometric shapes like "box", or "triangle", and then a higher layer might learn "nose" or "eye" based on those earlier features, and then a higher level layer still learns "face" made up from "eye", "nose", "jaw", etc. But even that, as I understand it, is still hypothetical and/or not understood in complete detail.
  
  > **Comments:**
  > **username_4**: I'd be interested to read the actual quotes. At the broadest conceptual level, the *why* is "They are Universal function approximators trained to reduce the error in a regression problem".  
  > **username_6**: I'll see if I can track them down.  I'm pretty sure the quote from Geoffrey Hinton that I'm thinking of is in a video.. either from his Coursera Class or some video he has up on Youtube.  If I can find it, I'll edit my answer and link it in.  
  > **username_6**: I haven't forgotten.  I'll try to find them when I have a little free time.   I think at least one of the ones I'm thinking of was from a video that's part of a Coursera course.  
  > **username_0**: This [study](https://www.cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf) can help to put same references: 'However there is no clear understanding of why they perform so well, or how they might be improved'.  
  
  ---
  
  ### Answer by username_7 on 2016-08-09. Score: 9
  
  Not sure if this is what you are searching for, but google extracted images from networks when they were fed with white noise.
  
  See [Inceptionism: Going Deeper into Neural Networks (Google Research Blog)](https://research.googleblog.com/2015/06/inceptionism-going-deeper-into-neural.html).
  
  This kind of represents what the network knows.
  
  
  ---
  
  ### Answer by username_8 on 2016-08-26. Score: 5
  
  Here is an answer by Carlos E. Perez to the question [What is theory behind deep learning?](https://www.quora.com/What-is-theory-behind-deep-learning/answer/Carlos-E-Perez?srid=CpS&share=0f965f46)
  
  > [...]
  >
  > The underlying mathematics of Deep Learning has been in existence for several decades, however the impressive results that we see today are part a consequence of much faster hardware, more data and incremental improvements in methods.
  >
  > Deep Learning in general can be framed as optimization problem where the objective is a function of the model error. This optimization problem is very difficult to solve consider that the parameter space of the model (i.e. weights of the neural network) leads to a problem in extremely high dimension. An optimization algorithm could take a very long time to explore this space. Furthermore, there was an unverified belief that the problem was non-convex and computation would forever be stuck in local minima.
  >
  > [...]
  >
  > The theory of why machines actually converge to an attractor or in other words learn to recognize complex patterns is still unknown.
  
  To sum up: we have some ideas, but we're not quite sure.
  
  
  ---
  
  ### Answer by username_9 on 2018-09-25. Score: 3
  
  > Do scientists know what is happening inside artificial neural networks?
  
  ### YES
  
  > Do scientists or research experts know from the kitchen what is happening inside complex "deep" neural network with at least millions of connections firing at an instant?
  
  I guess "to know from the kitchen" means "to know in detail"?
  
  Let me give you a series of analogies:
  
  1. Does an airplane engineer know from the kitchen what happens inside the airplane?
  2. Does a chip designer know in detail what happens in the chip (s)he designed?
  3. Does a civil engineer know everything about the house he constructed?
  
  The devil is in the detail, but a crucial point here is that it's about artificial structures. They don't randomly appear. You need a lot of knowledge to get anything useful. For Neural Networks, I would say it took roughly 40 years from the publication of the key idea (Rosenblatt perceptron, 1957) to the first application (US Postal Service, 1989). And from there again 13 years of active reserach to really impressive systems (ImageNet 2012).
  
  **What we know super well is how the training works**. Because it needs to be implemented. So on a very small structure, we know it in detail.
  
  Think of computers. The chip designers know very well how their chip works. But they will likely only have a very rough idea how the Linux operating system works.
  
  Another example is physics and chemistry: Physics describes the core forces of the universe. Does that mean they know everything about chemistry as well? Hell no! A "perfect" physicist can explain everything in chemistry ... but it would be pretty much useless. He would need a lot more information, not be able to skip the irrelevant parts. Simply because he "zoomed in" too much - considers details which are in practice neither interesting nor important. Please note that the knowledge of the physicist is not wrong. Maybe one could even deduce the knowledge from the chemist from it. But this "high-level" understanding of molecule interaction is missing.
  
  The key insight from those two examples are abstraction layers: **You can build complexity from simple structures**.
  
  What else?
  
  **We know well what is in principle achievable with the neural networks** we design:
  
  * A neural network designed to play Go - no matter how sophisticated - will never even be able to play chess. You can, of course, add another abstraction layer around it and combine things. But this approach needs humans.
  * A neural network designed for distinguishing dogs from cats which has only seen pudels and Persian cats will likely perform really bad when it has to decide for Yorkshire Terriers.
  
  Oh, and of course we have analytical approaches for neural networks. I wrote my masters thesis about [Analysis and Optimization of Convolutional Neural Network Architectures](https://arxiv.org/pdf/1707.09725.pdf). In this context [LIME](https://github.com/marcotcr/lime) (Local Interpretable Model-Agnostic Explanations) is nice:
  
  [![enter image description here](https://i.sstatic.net/wXzfn.png)](https://i.sstatic.net/wXzfn.png)
  
  > **Comments:**
  > **username_9**: Think of early day automobile / aircraft engineers. Would you say they don't know what's happening inside their aircraft / automobile, because they didn't built them, because their shape was not aerodynamic?  
  
  ---
  
  