Body: <p>Assume that we have 4 layers in a neural network.</p>
  <p><span class="math-container">$$z_1 = L_1(x, W_1)$$</span>
  <span class="math-container">$$z_2 = L_2(z_1, W_2)$$</span>
  <span class="math-container">$$z_3 = L_3(z_2, W_3)$$</span>
  <span class="math-container">$$y = L_1(z_3, W_4)$$</span></p>
  <p>Where <span class="math-container">$x$</span> is the vector input, <span class="math-container">$y$</span> is the vector output and <span class="math-container">$W_i, i = 1..4$</span> is the weight matrix.</p>
  <p>Assume that I could estimate parameters in a function.</p>
  <p><span class="math-container">$$b = f(a, w)$$</span></p>
  <p>Where the <span class="math-container">$b$</span> is a real value and <span class="math-container">$a$</span> is the input vector and <span class="math-container">$w$</span> is the weight vector parameter. The function <span class="math-container">$f$</span> could be like this.</p>
  <p><span class="math-container">$$b = \text{activation}(a_1*w_1 + a_2*w_2 + a_3*w_3 + \dots + a_n*w_n)$$</span></p>
  <p>Here we can interpret <span class="math-container">$b$</span> as the neuron output. Estimate <span class="math-container">$w_n$</span> is very easy if we know <span class="math-container">$b$</span> and <span class="math-container">$a_n$</span>. This can be done used by recursive least squares or a Kalman filter.</p>
  <p><strong>Question:</strong></p>
  <p>If every neuron in a neural network is a function that has inputs and weights, can I use parameter estimation for estimating all weights in a neural network if I did parameter estimation for every neuron inside a neural network?</p>
  <p><strong>The reason why I'm asking:</strong></p>
  <p>I found a <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.80.1421&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">paper</a> where they are using a Unscented Kalman Filter for parameter estimation.</p>
  <p>Function <span class="math-container">$D_{k|k-1} = G[x_k, W_{k|k-1}]$</span> can be interpreted as a neuron function where <span class="math-container">$W_{k|k-1}$</span> is a matrix with different types of weights and <span class="math-container">$D_{k|k-1}$</span> is different types of outputs from that neuron. No, it's not a &quot;multivariable output&quot;-neuron. It's just the way how to estimate the best weights by using different weights.</p>
  <p>The error of the neuron output is: <span class="math-container">$d_k - \hat d_k$</span> in equation (41).
  So when the error is small, that means the output of the neuron is OK and that means the real weights <span class="math-container">$\hat w_k$</span> has been found.</p>
  <p><a href="https://i.sstatic.net/WYqCxl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/WYqCxl.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T18:48:56.400Z
Comments.Text: I don't understand what you mean exactly by "parameter estimation" when you ask the question "can I use parameter estimation for estimating all weights in a neural network...?". Can you clarify that? It's not very clear the relationship between your idea of "estimating parameters" and the paper that you found.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T06:08:57.370Z
Comments.Text: @nbro I assume that parameter estimation is a tool for estimating unknown parameters inside an equation, by using data.
Comments.UserDisplayName: euraad
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T12:53:37.660Z
Comments.Text: So, would back-propagation (which is usually used to train neural networks) be a "parameter estimation" tool/technique or you have something else in mind?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T13:28:52.457Z
Comments.Text: @nbro No. Back-propagation is a optimization method. This method above is estimation. Back propagation you computing the weights by setting the outputs first. In this method above, we don't know the outputs. We only set the inputs and compute the weights just by looking how large the error is between the real data and estimated data.
Comments.UserDisplayName: euraad
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T13:30:02.433Z
Comments.Text: But back-propagation computes the gradient based on the error, which seems that it's what you're suggesting.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T13:31:35.390Z
Comments.Text: @nbro Yes. all algorithms need to have a reference. Gradient decent is using derivative and this algorithm above is using least squares. They both minimize the error.
Comments.UserDisplayName: euraad
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T13:34:59.877Z
Comments.Text: Ok, so what is your question then? If you're asking "Can we use back-propagation/gradient descent to train the neural network?", of course, the answer is generally yes. If you're interested in other specific training procedures, then you should probably be more specific. So, please, change your post and title to be more specific and clearer.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-20T14:21:25.980Z
Comments.Text: @nbro My question is very clear. I can change the title, it's good enough. I'm interesting if it's possible to estimate a neural network by using this method above, which is an estimation method by using a SR-UKF filter.
Comments.UserDisplayName: euraad
Tags: <neural-networks><weights><kalman-filter>
Title: Using parameter estimation for training a neural network
UserDisplayName: euraad
ThreadText: # Using parameter estimation for training a neural network
  
  (Asked by: username_0 on 2021-09-15)
  
  Assume that we have 4 layers in a neural network.
  
  $$z\_1 = L\_1(x, W\_1)$$
  $$z\_2 = L\_2(z\_1, W\_2)$$
  $$z\_3 = L\_3(z\_2, W\_3)$$
  $$y = L\_1(z\_3, W\_4)$$
  
  Where $x$ is the vector input, $y$ is the vector output and $W\_i, i = 1..4$ is the weight matrix.
  
  Assume that I could estimate parameters in a function.
  
  $$b = f(a, w)$$
  
  Where the $b$ is a real value and $a$ is the input vector and $w$ is the weight vector parameter. The function $f$ could be like this.
  
  $$b = \text{activation}(a\_1\*w\_1 + a\_2\*w\_2 + a\_3\*w\_3 + \dots + a\_n\*w\_n)$$
  
  Here we can interpret $b$ as the neuron output. Estimate $w\_n$ is very easy if we know $b$ and $a\_n$. This can be done used by recursive least squares or a Kalman filter.
  
  **Question:**
  
  If every neuron in a neural network is a function that has inputs and weights, can I use parameter estimation for estimating all weights in a neural network if I did parameter estimation for every neuron inside a neural network?
  
  **The reason why I'm asking:**
  
  I found a [paper](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.80.1421&rep=rep1&type=pdf) where they are using a Unscented Kalman Filter for parameter estimation.
  
  Function $D\_{k|k-1} = G[x\_k, W\_{k|k-1}]$ can be interpreted as a neuron function where $W\_{k|k-1}$ is a matrix with different types of weights and $D\_{k|k-1}$ is different types of outputs from that neuron. No, it's not a "multivariable output"-neuron. It's just the way how to estimate the best weights by using different weights.
  
  The error of the neuron output is: $d\_k - \hat d\_k$ in equation (41).
  So when the error is small, that means the output of the neuron is OK and that means the real weights $\hat w\_k$ has been found.
  
  [![enter image description here](https://i.sstatic.net/WYqCxl.png)](https://i.sstatic.net/WYqCxl.png)
  
  **Tags:** neural-networks, weights, kalman-filter
  
  > **Comments:**
  > **username_1**: I don't understand what you mean exactly by "parameter estimation" when you ask the question "can I use parameter estimation for estimating all weights in a neural network...?". Can you clarify that? It's not very clear the relationship between your idea of "estimating parameters" and the paper that you found.  
  > **username_0**: @username_1 I assume that parameter estimation is a tool for estimating unknown parameters inside an equation, by using data.  
  > **username_1**: So, would back-propagation (which is usually used to train neural networks) be a "parameter estimation" tool/technique or you have something else in mind?  
  > **username_0**: @username_1 No. Back-propagation is a optimization method. This method above is estimation. Back propagation you computing the weights by setting the outputs first. In this method above, we don't know the outputs. We only set the inputs and compute the weights just by looking how large the error is between the real data and estimated data.  
  > **username_1**: But back-propagation computes the gradient based on the error, which seems that it's what you're suggesting.  
  > **username_0**: @username_1 Yes. all algorithms need to have a reference. Gradient decent is using derivative and this algorithm above is using least squares. They both minimize the error.  
  > **username_1**: Ok, so what is your question then? If you're asking "Can we use back-propagation/gradient descent to train the neural network?", of course, the answer is generally yes. If you're interested in other specific training procedures, then you should probably be more specific. So, please, change your post and title to be more specific and clearer.  
  > **username_0**: @username_1 My question is very clear. I can change the title, it's good enough. I'm interesting if it's possible to estimate a neural network by using this method above, which is an estimation method by using a SR-UKF filter.  
  
  ---
  
  ## 0 answers
  
  