Answers.Body: <p>To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function <span class="math-container">$\mathbb{R}^n \rightarrow \mathbb{R}^m$</span> you need <span class="math-container">$nm$</span> weights aka the matrix corresponding to the linear application), with enough data, to 100% accuracy... then it is linear. </p>
  
  <p>The estimated function is explicit: it is given by the architecture of the NN and its weights. I don't think you can hope for an answer like "<span class="math-container">$\sin(\pi x + \phi) +4$</span>" if it's not already in the predictive capacity of the NN. Interpretability of NNs is a hot topic. In what cases symbolic reasoning can lead to a simplified expression?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-03T20:33:05.513Z
Answers.Text: You actually have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take a look at [this example](https://ai.stackexchange.com/a/25460/2444): i.e. it's just the expression that you use to perform the forward pass of the neural network. The problem is how to interpret this function, that's why neural networks are black-box models, because they are not easily interpretable.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-03T23:24:38.290Z
Answers.LastActivityDate: 2019-06-03T23:24:38.290Z
Answers.UserDisplayName: dcolazin
Answers.Body: <p>As opposed to what is written <a href="https://ai.stackexchange.com/a/12672/2444">in this answer</a>, you can have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take a look at <a href="https://ai.stackexchange.com/a/25460/2444">this example</a>: i.e. it is just the expression that you use to perform the forward pass of the neural network. The problem is how to interpret this function, how it generally behaves, and how it is different from the usually unknown function that the neural network is supposed to approximately compute: that's why neural networks are called <em>black-box</em> models, because they are not easily interpretable.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-03T20:36:56.073Z
Answers.LastActivityDate: 2021-01-03T20:36:56.073Z
Answers.UserDisplayName: nbro
Answers.Body: <p>The network is the function.</p>
  
  <p>A network is a function, that is modeled by terms describing the architecture and coefficients that are learned.</p>
  
  <p>Look at a simple model:</p>
  
  <p><span class="math-container">$$f(x) = ax+b$$</span></p>
  
  <p>Your solver determines <span class="math-container">$a$</span> and <span class="math-container">$b$</span>, and you substitute them into <span class="math-container">$f(x)$</span> and then you're able to calculate <span class="math-container">$f(42)$</span>. The function is linear by definition, but may not be a good fit for your data.</p>
  
  <p>To see if the input data may belong to a linear function can be achieved, when you have a network, that can both fit linear functions and higher order functions. For example try to fit your data to</p>
  
  <p><span class="math-container">$$f(x)=ax^2+bx+c$$</span></p>
  
  <p>The function is linear, when <span class="math-container">$a=0$</span>. If the function is quadratic you get <span class="math-container">$a\ne 0$</span> and if it has a higher order than <span class="math-container">$2$</span> you will get not good fit with any <span class="math-container">$a, b, c$</span>.</p>
  
  <p>When looking at MLP and similar networks, the question "is it linear" may not be that easy to answer, but the question how to get the function is the same: Your trained network is the function.</p>
  
  <p>I guess one option to see if the function approximated by an existing network is linear is to train a second network that only contains linear elements. If it is able to approximate your first network, the first network is linear as well. Of course you will need the right test data, as you may accidentally choose a training set that has linearly correlated output in a non-linear function. This is of course the same problem as always, that your network will always can only be as good as the training data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-05T11:29:00.503Z
Answers.LastActivityDate: 2019-06-07T08:13:15.883Z
Answers.LastEditDate: 2019-06-07T08:13:15.883Z
Answers.UserDisplayName: allo
Body: <p>As stated in the universal approximation theorem, a neural network can approximate almost any function.</p>
  <p>Is there a way to calculate the closed-form (or analytical) expression of the function that a neural network computes/approximates?</p>
  <p>Or, alternatively, figure out if the function is linear or non-linear?</p>
  
Tags: <neural-networks><function-approximation><universal-approximation-theorems>
Title: Is there a way to calculate the closed-form expression of the function that a neural network computes?
UserDisplayName: holistic
ThreadText: # Is there a way to calculate the closed-form expression of the function that a neural network computes?
  
  (Asked by: username_0 on 2019-06-03)
  
  As stated in the universal approximation theorem, a neural network can approximate almost any function.
  
  Is there a way to calculate the closed-form (or analytical) expression of the function that a neural network computes/approximates?
  
  Or, alternatively, figure out if the function is linear or non-linear?
  
  **Tags:** neural-networks, function-approximation, universal-approximation-theorems
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2019-06-03. Score: 1 | Accepted answer
  
  To check if a function is linear is easy: if you can train one fully connected layer, without activations, of the right dimensions (for a function $\mathbb{R}^n \rightarrow \mathbb{R}^m$ you need $nm$ weights aka the matrix corresponding to the linear application), with enough data, to 100% accuracy... then it is linear.
  
  The estimated function is explicit: it is given by the architecture of the NN and its weights. I don't think you can hope for an answer like "$\sin(\pi x + \phi) +4$" if it's not already in the predictive capacity of the NN. Interpretability of NNs is a hot topic. In what cases symbolic reasoning can lead to a simplified expression?
  
  > **Comments:**
  > **username_2**: You actually have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take a look at [this example](https://ai.stackexchange.com/a/25460/2444): i.e. it's just the expression that you use to perform the forward pass of the neural network. The problem is how to interpret this function, that's why neural networks are black-box models, because they are not easily interpretable.  
  
  ---
  
  ### Answer by username_2 on 2021-01-03. Score: 1
  
  As opposed to what is written [in this answer](https://ai.stackexchange.com/a/12672/2444), you can have the analytical expression of the function that the neural network computes, even if that neural network computes a non-linear function. Take a look at [this example](https://ai.stackexchange.com/a/25460/2444): i.e. it is just the expression that you use to perform the forward pass of the neural network. The problem is how to interpret this function, how it generally behaves, and how it is different from the usually unknown function that the neural network is supposed to approximately compute: that's why neural networks are called *black-box* models, because they are not easily interpretable.
  
  
  ---
  
  ### Answer by username_3 on 2019-06-05. Score: 1
  
  The network is the function.
  
  A network is a function, that is modeled by terms describing the architecture and coefficients that are learned.
  
  Look at a simple model:
  
  $$f(x) = ax+b$$
  
  Your solver determines $a$ and $b$, and you substitute them into $f(x)$ and then you're able to calculate $f(42)$. The function is linear by definition, but may not be a good fit for your data.
  
  To see if the input data may belong to a linear function can be achieved, when you have a network, that can both fit linear functions and higher order functions. For example try to fit your data to
  
  $$f(x)=ax^2+bx+c$$
  
  The function is linear, when $a=0$. If the function is quadratic you get $a\ne 0$ and if it has a higher order than $2$ you will get not good fit with any $a, b, c$.
  
  When looking at MLP and similar networks, the question "is it linear" may not be that easy to answer, but the question how to get the function is the same: Your trained network is the function.
  
  I guess one option to see if the function approximated by an existing network is linear is to train a second network that only contains linear elements. If it is able to approximate your first network, the first network is linear as well. Of course you will need the right test data, as you may accidentally choose a training set that has linearly correlated output in a non-linear function. This is of course the same problem as always, that your network will always can only be as good as the training data.
  
  
  ---
  
  