Answers.Body: <p>&quot;<em>Ideally, I would end up with an equation that would allow me to perform the classification without the network</em>&quot;.</p>
  <p>If you could find such analytic equation without machine learning then why training a multi layer perceptron in the first place? Or to phrase it differently, the mlp you trained is that equation. And I'm not trying to be ironic, if you need an analytic explanation then don't use a multi layer perceptron, but move to decision trees algorithms for example, than you could literally plot the model itself (still hard to interpret depending on the amount of features you're using).</p>
  <p>If instead you still want to stick with mlp, then something that you could do to better understand your model is plotting the decision boundaries learned by it. Sklearn has a nice <a href="https://scikit-learn.org/0.18/auto_examples/svm/plot_iris.html" rel="nofollow noreferrer">tutorial</a> on how to do it, I copied it and changes the svm with an mlp just to show that the approach works regardless of the model</p>
  <pre><code>import numpy as np
  import matplotlib.pyplot as plt
  from sklearn import datasets
  from sklearn.neural_network import MLPClassifier
  
  # import some data to play with
  iris = datasets.load_iris()
  X = iris.data[:, :2]  # we only take the first two features. We could
  # avoid this ugly slicing by using a two-dim dataset
  y = iris.target
  
  h = 0.02  # step size in the mesh
  
  # we create an instance of SVM and fit out data. We do not scale our
  # data since we want to plot the support vectors
  C = 1.0  # SVM regularization parameter
  mlp = MLPClassifier().fit(X, y)
  
  # create a mesh to plot in
  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
  
  # title for the plots
  title = &quot;MLP boundries&quot;
  
  
  # Plot the decision boundary. For that, we will assign a color to each
  # point in the mesh [x_min, x_max]x[y_min, y_max].
  # plt.subplot(1, 1)
  # plt.subplots_adjust(wspace=0.4, hspace=0.4)
  
  Z = mlp.predict(np.c_[xx.ravel(), yy.ravel()])
  
  # Put the result into a color plot
  Z = Z.reshape(xx.shape)
  plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)
  
  # Plot also the training points
  plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm)
  plt.xlabel(&quot;Sepal length&quot;)
  plt.ylabel(&quot;Sepal width&quot;)
  plt.xlim(xx.min(), xx.max())
  plt.ylim(yy.min(), yy.max())
  plt.xticks(())
  plt.yticks(())
  plt.title(title)
  
  plt.show()
  </code></pre>
  <p>The code output this plot:
  <a href="https://i.sstatic.net/s22zK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/s22zK.png" alt="enter image description here" /></a></p>
  <p>Your can see that the boundaries are a crude approximation of the mlp behavior, they are just estimated from a brute force prediction applied to all points of the 2D graph generated by 2 features of the input data. SO the boundaries will change depending also on the features you decide to plot. But it gives you an idea about the relationships learned by the mlp.</p>
  <p>If you want more, I stress again that you should train a different model, like decision tree, random forest or XGboost, with these model you can compute scores about features importance and literally plot the decision thresholds learned by the models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-20T10:11:40.187Z
Answers.LastActivityDate: 2021-08-20T14:01:49.493Z
Answers.LastEditDate: 2021-08-20T14:01:49.493Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem.</p>
  <p>The network works well, but I'd like to know how it is using the inputs to perform the classification. Ultimately, I would like to interpret the trained network in order to learn more about the processes responsible for generating the data.</p>
  <p>Ideally, I would end up with an equation that would allow me to perform the classification without the network and that would have parameters that I could interpret in the context of the system the network is being used on.</p>
  <p>My first thought is to procedurally mask out a growing subset of the ~4000 parameters until there's an appropriate trade-off between performance and simplicity and then maybe use a symbolic logic library to try and simplify further.</p>
  <p>I don't think that's the best plan, so I wonder if there's an existing workflow to interpret a neural network.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T14:17:25.410Z
Comments.Text: It's not clear to me what you're really asking here. Are you asking how to represent this specific neural network as an equation, so something like $f(x) = 2x^2 + 3$?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T23:00:45.563Z
Comments.Text: @nbro I've edited the question to hopefully make it less vague. The data were generated from a simulation of a biological process that I would like to learn more about. I would like to use not the architecture, but the trained model to ideally generate an equation that relates 15 inputs to one binary output.
Comments.UserDisplayName: asheets
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T23:04:12.367Z
Comments.Text: Thanks! Your last edit clarifies what you are asking. I tried to make the title the question you're asking. Make sure that's the case. If not, feel free to edit the post again.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-19T23:05:49.753Z
Comments.Text: Having said that, the answer to your question may be in one of the answers to [this question](https://ai.stackexchange.com/q/12870/2444).  I will not mark yours as a duplicate, because it seems to me that your problem/question is more specific.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-20T15:39:12.840Z
Comments.Text: If you're using something like PyTorch or TensorFlow, you can also create synthetic samples using gradient feedback loops which creates saliency maps showing what features really might help triggering your classification. See this: https://github.com/utkuozbulak/pytorch-cnn-visualizations
Comments.UserDisplayName: Farhood ET
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-11-03T14:16:41.710Z
Comments.Text: I would suggest looking into feature attributions for this, e.g. by using the captum library: https://captum.ai/
Comments.UserDisplayName: jumelet
Tags: <neural-networks><classification><explainable-ai>
Title: How can I interpret the way the neural network is producing an output for a given input?
UserDisplayName: asheets
ThreadText: # How can I interpret the way the neural network is producing an output for a given input?
  
  (Asked by: username_0 on 2021-08-19)
  
  I'm using a small neural network (2 hidden layers, 60 neurons apiece) for a rather complex binary classification problem.
  
  The network works well, but I'd like to know how it is using the inputs to perform the classification. Ultimately, I would like to interpret the trained network in order to learn more about the processes responsible for generating the data.
  
  Ideally, I would end up with an equation that would allow me to perform the classification without the network and that would have parameters that I could interpret in the context of the system the network is being used on.
  
  My first thought is to procedurally mask out a growing subset of the ~4000 parameters until there's an appropriate trade-off between performance and simplicity and then maybe use a symbolic logic library to try and simplify further.
  
  I don't think that's the best plan, so I wonder if there's an existing workflow to interpret a neural network.
  
  **Tags:** neural-networks, classification, explainable-ai
  
  > **Comments:**
  > **username_1**: It's not clear to me what you're really asking here. Are you asking how to represent this specific neural network as an equation, so something like $f(x) = 2x^2 + 3$?  
  > **username_0**: @username_1 I've edited the question to hopefully make it less vague. The data were generated from a simulation of a biological process that I would like to learn more about. I would like to use not the architecture, but the trained model to ideally generate an equation that relates 15 inputs to one binary output.  
  > **username_1**: Thanks! Your last edit clarifies what you are asking. I tried to make the title the question you're asking. Make sure that's the case. If not, feel free to edit the post again.  
  > **username_1**: Having said that, the answer to your question may be in one of the answers to [this question](https://ai.stackexchange.com/q/12870/2444).  I will not mark yours as a duplicate, because it seems to me that your problem/question is more specific.  
  > **username_2**: If you're using something like PyTorch or TensorFlow, you can also create synthetic samples using gradient feedback loops which creates saliency maps showing what features really might help triggering your classification. See this: https://github.com/utkuozbulak/pytorch-cnn-visualizations  
  > **username_3**: I would suggest looking into feature attributions for this, e.g. by using the captum library: https://captum.ai/  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2021-08-20. Score: 2
  
  "*Ideally, I would end up with an equation that would allow me to perform the classification without the network*".
  
  If you could find such analytic equation without machine learning then why training a multi layer perceptron in the first place? Or to phrase it differently, the mlp you trained is that equation. And I'm not trying to be ironic, if you need an analytic explanation then don't use a multi layer perceptron, but move to decision trees algorithms for example, than you could literally plot the model itself (still hard to interpret depending on the amount of features you're using).
  
  If instead you still want to stick with mlp, then something that you could do to better understand your model is plotting the decision boundaries learned by it. Sklearn has a nice [tutorial](https://scikit-learn.org/0.18/auto_examples/svm/plot_iris.html) on how to do it, I copied it and changes the svm with an mlp just to show that the approach works regardless of the model
  
  ```
  import numpy as np
  import matplotlib.pyplot as plt
  from sklearn import datasets
  from sklearn.neural_network import MLPClassifier
  
  # import some data to play with
  iris = datasets.load_iris()
  X = iris.data[:, :2]  # we only take the first two features. We could
  # avoid this ugly slicing by using a two-dim dataset
  y = iris.target
  
  h = 0.02  # step size in the mesh
  
  # we create an instance of SVM and fit out data. We do not scale our
  # data since we want to plot the support vectors
  C = 1.0  # SVM regularization parameter
  mlp = MLPClassifier().fit(X, y)
  
  # create a mesh to plot in
  x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
  y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
  xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
  
  # title for the plots
  title = "MLP boundries"
  
  # Plot the decision boundary. For that, we will assign a color to each
  # point in the mesh [x_min, x_max]x[y_min, y_max].
  # plt.subplot(1, 1)
  # plt.subplots_adjust(wspace=0.4, hspace=0.4)
  
  Z = mlp.predict(np.c_[xx.ravel(), yy.ravel()])
  
  # Put the result into a color plot
  Z = Z.reshape(xx.shape)
  plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)
  
  # Plot also the training points
  plt.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.coolwarm)
  plt.xlabel("Sepal length")
  plt.ylabel("Sepal width")
  plt.xlim(xx.min(), xx.max())
  plt.ylim(yy.min(), yy.max())
  plt.xticks(())
  plt.yticks(())
  plt.title(title)
  
  plt.show()
  
  ```
  
  The code output this plot:
  [![enter image description here](https://i.sstatic.net/s22zK.png)](https://i.sstatic.net/s22zK.png)
  
  Your can see that the boundaries are a crude approximation of the mlp behavior, they are just estimated from a brute force prediction applied to all points of the 2D graph generated by 2 features of the input data. SO the boundaries will change depending also on the features you decide to plot. But it gives you an idea about the relationships learned by the mlp.
  
  If you want more, I stress again that you should train a different model, like decision tree, random forest or XGboost, with these model you can compute scores about features importance and literally plot the decision thresholds learned by the models.
  
  
  ---
  
  