Answers.Body: <p>RLS is a second order optimizer, so, unlike LMS which takes into account an approximation of the derivative of the gradient, RLS also considers the second order derivative. You can study more about second order methods in sub-section &quot;8.6 Approximate Second-Order Methods&quot; of the following book available online:</p>
  <p><a href="https://www.deeplearningbook.org/contents/optimization.html" rel="nofollow noreferrer">https://www.deeplearningbook.org/contents/optimization.html</a></p>
  <p>Deep Learning
  An MIT Press book
  Ian Goodfellow and Yoshua Bengio and Aaron Courville</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-18T17:14:59.853Z
Answers.LastActivityDate: 2020-11-18T17:14:59.853Z
Answers.UserDisplayName: F4RZ4D
Body: <p>I'm new to neural network, I study electrical engineering, and I just started working with <a href="https://en.wikipedia.org/wiki/ADALINE#Learning_algorithm" rel="nofollow noreferrer">ADALINEs</a>.</p>
  
  <p>I use Matlab, and in their <a href="https://www.mathworks.com/help/nnet/ug/adaptive-neural-network-filters.html" rel="nofollow noreferrer">Documentation</a> they cite :</p>
  
  <blockquote>
    <p>However, here the <strong>LMS (least mean squares) learning rule</strong>, which is
    much more powerful than the perceptron learning rule, is used. The
    LMS, or Widrow-Hoff, learning rule minimizes the mean square error and
    thus moves the decision boundaries as far as it can from the training
    patterns.</p>
  </blockquote>
  
  <p>The LMS algorithm is the default learning rule to linear neural network in Matlab, but a few days later I came across another algorithm which is : <a href="https://en.wikipedia.org/wiki/Recursive_least_squares_filter" rel="nofollow noreferrer">Recursive Least Squares (RLS)</a> in a 2017 Research Article by <strong>Sachin Devassy</strong> and  <strong>Bhim Singh</strong> in the journal: <strong>IET Renewable Power Generation</strong>, under the title : <a href="http://ieeexplore.ieee.org/document/8049622/" rel="nofollow noreferrer">Performance analysis of proportional resonant and ADALINE-based solar photovoltaic-integrated unified active power filter</a> where they state :</p>
  
  <blockquote>
    <p>ADALINE-based approach is an efficient method for extracting
    fundamental component of load active current as no additional
    transformation and inverse transformations are required. The various
    adaptation algorithms include <strong>least mean square</strong>, <strong>recursive least</strong>
    <strong>squares</strong> etc.</p>
  </blockquote>
  
  <p><strong>My questions are:</strong></p>
  
  <ul>
  <li>Is RLS just like LMS (i mean can it be used as a learning algorithm too)? </li>
  <li>If yes, how can I customize my ADALINE to use RLS instead of LMS as a learning algorithm (preferably in Matlab, if not in Python) because I want to do a comparative study between the two Algorithm!</li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-17T09:39:49.790Z
Comments.Text: Really you want use an Adlines NN instead of perceptrons ? Sorry for the a few off-topic comment, but surprises me.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-17T09:42:30.467Z
Comments.Text: LMS and RLS are not learning rules, but cost functions. RLS th same than LMS after apply a filter to the error measures, usually a forget factor.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-17T09:44:23.157Z
Comments.Text: Please feel free to comment, i'm new so honestly i don't know, if there is something you want to suggest i'm listening, thank you
Comments.UserDisplayName: Carter Nolan
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2018-03-17T10:42:10.450Z
Comments.Text: If there are not a non-linear element we are more in the area of linear filtering than in the one of AI. Moreover, as combination of linear elements can be flatten, multi-layer concept disappears. Non-linear activation functions as Heaviside (perceptron), sigmoid, ... are the center of the NN.
Comments.UserDisplayName: pasaba por aqui
Tags: <learning-algorithms><linear-regression>
Title: Can we use the recursive least squares as a learning algorithm to an ADALINE?
UserDisplayName: Carter Nolan
ThreadText: # Can we use the recursive least squares as a learning algorithm to an ADALINE?
  
  (Asked by: username_0 on 2018-03-17)
  
  I'm new to neural network, I study electrical engineering, and I just started working with [ADALINEs](https://en.wikipedia.org/wiki/ADALINE#Learning_algorithm).
  
  I use Matlab, and in their [Documentation](https://www.mathworks.com/help/nnet/ug/adaptive-neural-network-filters.html) they cite :
  
  > However, here the **LMS (least mean squares) learning rule**, which is
  > much more powerful than the perceptron learning rule, is used. The
  > LMS, or Widrow-Hoff, learning rule minimizes the mean square error and
  > thus moves the decision boundaries as far as it can from the training
  > patterns.
  
  The LMS algorithm is the default learning rule to linear neural network in Matlab, but a few days later I came across another algorithm which is : [Recursive Least Squares (RLS)](https://en.wikipedia.org/wiki/Recursive_least_squares_filter) in a 2017 Research Article by **Sachin Devassy** and **Bhim Singh** in the journal: **IET Renewable Power Generation**, under the title : [Performance analysis of proportional resonant and ADALINE-based solar photovoltaic-integrated unified active power filter](http://ieeexplore.ieee.org/document/8049622/) where they state :
  
  > ADALINE-based approach is an efficient method for extracting
  > fundamental component of load active current as no additional
  > transformation and inverse transformations are required. The various
  > adaptation algorithms include **least mean square**, **recursive least**
  > **squares** etc.
  
  **My questions are:**
  
  * Is RLS just like LMS (i mean can it be used as a learning algorithm too)?
  * If yes, how can I customize my ADALINE to use RLS instead of LMS as a learning algorithm (preferably in Matlab, if not in Python) because I want to do a comparative study between the two Algorithm!
  
  **Tags:** learning-algorithms, linear-regression
  
  > **Comments:**
  > **username_1**: Really you want use an Adlines NN instead of perceptrons ? Sorry for the a few off-topic comment, but surprises me.  
  > **username_1**: LMS and RLS are not learning rules, but cost functions. RLS th same than LMS after apply a filter to the error measures, usually a forget factor.  
  > **username_0**: Please feel free to comment, i'm new so honestly i don't know, if there is something you want to suggest i'm listening, thank you  
  > **username_1**: If there are not a non-linear element we are more in the area of linear filtering than in the one of AI. Moreover, as combination of linear elements can be flatten, multi-layer concept disappears. Non-linear activation functions as Heaviside (perceptron), sigmoid, ... are the center of the NN.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-18. Score: 1
  
  RLS is a second order optimizer, so, unlike LMS which takes into account an approximation of the derivative of the gradient, RLS also considers the second order derivative. You can study more about second order methods in sub-section "8.6 Approximate Second-Order Methods" of the following book available online:
  
  <https://www.deeplearningbook.org/contents/optimization.html>
  
  Deep Learning
  An MIT Press book
  Ian Goodfellow and Yoshua Bengio and Aaron Courville
  
  
  ---
  
  