Body: <p>I found a <a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.80.1421&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">paper</a> about using an Unscented Kalman Filter(UKF) for traning a neural network.</p>
  <p>The UKF filter is modified so it works for parameter estimation. Assume that we have a neural network model <span class="math-container">$\hat d_k = G(x_k, W_k)$</span> where <span class="math-container">$G$</span> is the neural network, <span class="math-container">$x_k$</span> is the input <strong>vector</strong>, <span class="math-container">$W_k$</span> is the parameter gain <strong>matrix</strong> and <span class="math-container">$\hat d_k$</span> is the output <strong>vector</strong>.</p>
  <p>This paper counts servral methods to train a neural network.</p>
  <ul>
  <li>Gradient descent</li>
  <li>Quasi-Newton</li>
  <li>UKF parameter estimation</li>
  </ul>
  <p><strong>So my questions for you are:</strong></p>
  <ol>
  <li><p>What's the benefit for using a kalman filter for training a neural network compared to other optimization algorithms?</p>
  </li>
  <li><p>I remember that I have been using gradient descent methods and they work, but I have always used regularization. Is that due to the noise in the data?</p>
  </li>
  <li><p>If I'm using a UKF-filter as parameter estimation, I can avoid regularization then?</p>
  </li>
  <li><p>Is this UKF parameter estimation algorithm only made for 1 layer neural network, or can it be used for deep neural networks as well?</p>
  </li>
  </ol>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-14T21:53:17.000Z
Comments.Text: Hello, asking multiple questions in a single post is not recommended. Please try to split the questions into multiple posts.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-15T22:41:22.227Z
Comments.Text: @hanugm Yes! I created a new question with only 1 question. https://ai.stackexchange.com/questions/31701/using-parameter-estimation-for-training-a-neural-network
Comments.UserDisplayName: euraad
Tags: <neural-networks><training><kalman-filter>
Title: What's the benefit for using a Kalman filter for training a neural network compared to other optimization algorithms?
UserDisplayName: euraad
ThreadText: # What's the benefit for using a Kalman filter for training a neural network compared to other optimization algorithms?
  
  (Asked by: username_0 on 2021-09-14)
  
  I found a [paper](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.80.1421&rep=rep1&type=pdf) about using an Unscented Kalman Filter(UKF) for traning a neural network.
  
  The UKF filter is modified so it works for parameter estimation. Assume that we have a neural network model $\hat d\_k = G(x\_k, W\_k)$ where $G$ is the neural network, $x\_k$ is the input **vector**, $W\_k$ is the parameter gain **matrix** and $\hat d\_k$ is the output **vector**.
  
  This paper counts servral methods to train a neural network.
  
  * Gradient descent
  * Quasi-Newton
  * UKF parameter estimation
  
  **So my questions for you are:**
  
  1. What's the benefit for using a kalman filter for training a neural network compared to other optimization algorithms?
  2. I remember that I have been using gradient descent methods and they work, but I have always used regularization. Is that due to the noise in the data?
  3. If I'm using a UKF-filter as parameter estimation, I can avoid regularization then?
  4. Is this UKF parameter estimation algorithm only made for 1 layer neural network, or can it be used for deep neural networks as well?
  
  **Tags:** neural-networks, training, kalman-filter
  > **Comments:**
  > **username_1**: Hello, asking multiple questions in a single post is not recommended. Please try to split the questions into multiple posts.  
  > **username_0**: @hanugm Yes! I created a new question with only 1 question. https://ai.stackexchange.com/questions/31701/using-parameter-estimation-for-training-a-neural-network  
  
  ---
  
  ## 0 answers
  
  