Body: <p>When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world.  Although different data sets or even different runs of the same NN might yield completely different sets of weights, the resulting equations must be mathematically similar (linear combinations, rotations, etc.).  Since we usually build NNs to model a particular concrete task (identify cats, pedestrians, tumors, etc.), it seems that we are generally satisfied to let the network continue to act as a black box.</p>
  
  <p>Now, I understand that there is a push for "<a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="noreferrer">understandability</a>" of NNs, other ML techniques, etc.  But this is not quite what I'm getting at.  It seems to me that given a bunch of data points recording the behavior of charged particles, one could effectively recover Maxwell's laws using a sufficiently advanced NN.  Perhaps that requires NNs which are much more sophisticated than what we have today.  But it illustrates the thing I <em>am</em> interested in: NNs could, in my mind, be teaching us general truths about the world if we took the time to analyze and simplify the formulae that they give us<sup><a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="noreferrer">1</a></sup>.</p>
  
  <p>For instance, there must be hundreds, if not thousands of NNs which have been trained on visual recognition tasks that end up learning many of the same sub-skills, to put it a bit anthropomorphically.  I recently read about <a href="https://towardsdatascience.com/an-easy-guide-to-gauge-equivariant-convolutional-networks-9366fb600b70" rel="noreferrer">gauge CNNs</a>, but this goes the other way: we start with what we know and then bake it into the network.</p>
  
  <p>Has anyone attempted to go the opposite way?  Either:</p>
  
  <ol>
  <li>Take a bunch of similar NNs and analyze what they have in common to extract general formulae about the focus area<sup><a href="https://towardsdatascience.com/an-easy-guide-to-gauge-equivariant-convolutional-networks-9366fb600b70" rel="noreferrer">2</a></sup></li>
  <li>Carefully inspect the structure of a well-trained NN to directly extract the "Maxwell's equations" which might be hiding in them?</li>
  </ol>
  
  <p><a href="https://en.wikipedia.org/wiki/Explainable_artificial_intelligence" rel="noreferrer">1</a> Imagine if we built a NN to learn Newtonian mechanics just to compute a simple ballistic trajectory.  It could surely be done, but would also be a massive waste of resources.  We have nice, neat equations for ballistic motion, given to us by the "original neural networks", so we use those.</p>
  
  <p><a href="https://towardsdatascience.com/an-easy-guide-to-gauge-equivariant-convolutional-networks-9366fb600b70" rel="noreferrer">2</a> E.g., surely the set of all visual NNs have collectively discovered near-optimal algorithms for edge/line/orientation detection, etc.).  This could perhaps be done with the assistance of a meta-ML algorithm (like, clustering over NN weight matrices?).</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-09T23:34:15.390Z
Comments.Text: Hi and welcome to this community! You say "Carefully inspect the structure of a well-trained NN to directly extract the "Maxwell's equations", but a neural network represents a function (or a set of functions). How do you plan to transform a function into an equation? Also, some neural networks have thousands or millions of parameters. How do you suggest inspecting the structure of such a big (trained) neural network? Furthermore, can you clarify how this is different than XAI?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-10T00:43:23.323Z
Comments.Text: My understanding of XAI is that it's like the SQL EXPLAIN PLAN for NNs.  "Justify your answer, but don't give me the full theory."  Whereas, extracting the approximated function would be like discovering the query engine itself.
  
  How would you rediscover Newton's laws? You might say: "Ok, we've got some observational data on moving objects with known masses, locations and velocities.  Let's make the network predict accelerations or forces or momenta." (or permute the observables) Do something analogous for other "laws".
  
  As for "how"? That's what the experts are for, right? ;)
Comments.UserDisplayName: Lawnmower Man
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-10T00:47:47.267Z
Comments.Text: To be more precise, XAI suggests to me that we can rationalize a *particular* (input, output) pair, as opposed to describing an over-arching framework which could effectively replace the NN itself as a computational shortcut (in exactly the same way as Newton's laws could replace a "ballistic NN").  As for analyzing overly large networks, we have ML tools to help us discover patterns in such things, no? If I could tell you exactly how to do it, I'd just do it myself. ;)
Comments.UserDisplayName: Lawnmower Man
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-10T01:13:46.090Z
Comments.Text: It's still unclear what you're looking for. You say "As for analyzing overly large networks, we have ML tools to help us discover patterns in such things, no?", so do you want to find encoded equations in the structure of a trained NN with another pattern recognition model? Or are you looking for an artificial scientist? Then, no, there is no artificial scientist (AFAIK). As I said, neural networks represent functions and not equations. The closest thing to what you're talking about that I recall is the [neural programmer-interpreter](https://arxiv.org/abs/1511.06279) (and similar stuff).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-10T03:01:51.990Z
Comments.Text: Every NN approximates some function, but only represents said function implicitly. The first step would be to develop a process which discovers an explicit, algebraic/analytical form for said function, perhaps by starting with simple candidates and iteratively enriching them until they converge on the NN's representation. Obviously, the hyperplane representing the ordinary least squares regression of the NN function would be the simplest starting point, and one could add higher-order terms until they stopped improving the fit.
Comments.UserDisplayName: Lawnmower Man
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-01-10T03:06:16.883Z
Comments.Text: However, "interesting" NNs likely represent a very complicated function which is the composition of much simpler "component" functions, and trying to infer such components may be possible with other ML techniques (like PCA, just to throw something out there...not sure how to directly apply it to this problem). Such a *process* is not an 'artificial scientist', because it doesn't uniquely identify a "simplest form". Humans using the tools would use their professional judgment to say: "Ah-hah! I spot an elegant formula embedded in this function!" Just as data scientists work their craft.
Comments.UserDisplayName: Lawnmower Man
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-21T12:38:37.633Z
Comments.Text: After several months, I came across this question again. I think that you're asking two distinct questions here: 1. "How to extract the commonalities that different neural networks may have?" and 2. "How to extract the explicit or symbolic representation of the function that a neural network represents?". These two questions are quite distinct, although somehow related. Despite that, I suggest that you edit your post to **ask only one question** here. Remove one of the two: the one that you don't like less. Then maybe create another post to ask the second question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-06-21T12:40:52.340Z
Comments.Text: I provisionally closed this post until you focus on one of the questions. I will re-open it after you have followed my suggestion above.
Comments.UserDisplayName: nbro
Tags: <neural-networks><deep-learning><explainable-ai>
Title: Has anyone attempted to take a bunch of similar neural networks to extract general formulae about the focus area?
UserDisplayName: Lawnmower Man
ThreadText: # Has anyone attempted to take a bunch of similar neural networks to extract general formulae about the focus area?
  
  (Asked by: username_0 on 2020-01-09)
  
  When a neural network learns something from a data set, we are left with a bunch of weights which represent some approximation of knowledge about the world. Although different data sets or even different runs of the same NN might yield completely different sets of weights, the resulting equations must be mathematically similar (linear combinations, rotations, etc.). Since we usually build NNs to model a particular concrete task (identify cats, pedestrians, tumors, etc.), it seems that we are generally satisfied to let the network continue to act as a black box.
  
  Now, I understand that there is a push for "[understandability](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence)" of NNs, other ML techniques, etc. But this is not quite what I'm getting at. It seems to me that given a bunch of data points recording the behavior of charged particles, one could effectively recover Maxwell's laws using a sufficiently advanced NN. Perhaps that requires NNs which are much more sophisticated than what we have today. But it illustrates the thing I *am* interested in: NNs could, in my mind, be teaching us general truths about the world if we took the time to analyze and simplify the formulae that they give us[1](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence).
  
  For instance, there must be hundreds, if not thousands of NNs which have been trained on visual recognition tasks that end up learning many of the same sub-skills, to put it a bit anthropomorphically. I recently read about [gauge CNNs](https://towardsdatascience.com/an-easy-guide-to-gauge-equivariant-convolutional-networks-9366fb600b70), but this goes the other way: we start with what we know and then bake it into the network.
  
  Has anyone attempted to go the opposite way? Either:
  
  1. Take a bunch of similar NNs and analyze what they have in common to extract general formulae about the focus area[2](https://towardsdatascience.com/an-easy-guide-to-gauge-equivariant-convolutional-networks-9366fb600b70)
  2. Carefully inspect the structure of a well-trained NN to directly extract the "Maxwell's equations" which might be hiding in them?
  
  [1](https://en.wikipedia.org/wiki/Explainable_artificial_intelligence) Imagine if we built a NN to learn Newtonian mechanics just to compute a simple ballistic trajectory. It could surely be done, but would also be a massive waste of resources. We have nice, neat equations for ballistic motion, given to us by the "original neural networks", so we use those.
  
  [2](https://towardsdatascience.com/an-easy-guide-to-gauge-equivariant-convolutional-networks-9366fb600b70) E.g., surely the set of all visual NNs have collectively discovered near-optimal algorithms for edge/line/orientation detection, etc.). This could perhaps be done with the assistance of a meta-ML algorithm (like, clustering over NN weight matrices?).
  
  **Tags:** neural-networks, deep-learning, explainable-ai
  
  > **Comments:**
  > **username_1**: Hi and welcome to this community! You say "Carefully inspect the structure of a well-trained NN to directly extract the "Maxwell's equations", but a neural network represents a function (or a set of functions). How do you plan to transform a function into an equation? Also, some neural networks have thousands or millions of parameters. How do you suggest inspecting the structure of such a big (trained) neural network? Furthermore, can you clarify how this is different than XAI?  
  > **username_0**: My understanding of XAI is that it's like the SQL EXPLAIN PLAN for NNs.  "Justify your answer, but don't give me the full theory."  Whereas, extracting the approximated function would be like discovering the query engine itself.  How would you rediscover Newton's laws? You might say: "Ok, we've got some observational data on moving objects with known masses, locations and velocities.  Let's make the network predict accelerations or forces or momenta." (or permute the observables) Do something analogous for other "laws".  As for "how"? That's what the experts are for, right? ;)  
  > **username_0**: To be more precise, XAI suggests to me that we can rationalize a *particular* (input, output) pair, as opposed to describing an over-arching framework which could effectively replace the NN itself as a computational shortcut (in exactly the same way as Newton's laws could replace a "ballistic NN").  As for analyzing overly large networks, we have ML tools to help us discover patterns in such things, no? If I could tell you exactly how to do it, I'd just do it myself. ;)  
  > **username_1**: It's still unclear what you're looking for. You say "As for analyzing overly large networks, we have ML tools to help us discover patterns in such things, no?", so do you want to find encoded equations in the structure of a trained NN with another pattern recognition model? Or are you looking for an artificial scientist? Then, no, there is no artificial scientist (AFAIK). As I said, neural networks represent functions and not equations. The closest thing to what you're talking about that I recall is the [neural programmer-interpreter](https://arxiv.org/abs/1511.06279) (and similar stuff).  
  > **username_0**: Every NN approximates some function, but only represents said function implicitly. The first step would be to develop a process which discovers an explicit, algebraic/analytical form for said function, perhaps by starting with simple candidates and iteratively enriching them until they converge on the NN's representation. Obviously, the hyperplane representing the ordinary least squares regression of the NN function would be the simplest starting point, and one could add higher-order terms until they stopped improving the fit.  
  > **username_0**: However, "interesting" NNs likely represent a very complicated function which is the composition of much simpler "component" functions, and trying to infer such components may be possible with other ML techniques (like PCA, just to throw something out there...not sure how to directly apply it to this problem). Such a *process* is not an 'artificial scientist', because it doesn't uniquely identify a "simplest form". Humans using the tools would use their professional judgment to say: "Ah-hah! I spot an elegant formula embedded in this function!" Just as data scientists work their craft.  
  > **username_1**: After several months, I came across this question again. I think that you're asking two distinct questions here: 1. "How to extract the commonalities that different neural networks may have?" and 2. "How to extract the explicit or symbolic representation of the function that a neural network represents?". These two questions are quite distinct, although somehow related. Despite that, I suggest that you edit your post to **ask only one question** here. Remove one of the two: the one that you don't like less. Then maybe create another post to ask the second question.  
  > **username_1**: I provisionally closed this post until you focus on one of the questions. I will re-open it after you have followed my suggestion above.  
  
  ---
  
  ## 0 answers
  
  