Answers.Body: <p><a href="https://en.wikipedia.org/wiki/Gradient_descent" rel="noreferrer"><strong>Gradient descent</strong></a> (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a <a href="https://en.wikipedia.org/wiki/Function_of_several_real_variables" rel="noreferrer">multi-variable</a> and <a href="https://en.wikipedia.org/wiki/Differentiable_function" rel="noreferrer">differentiable</a> function <span class="math-container">$f$</span>. GD is an <em>iterative</em> and <em>numerical</em> optimisation algorithm. It is <em>iterative</em> because it proceeds in iterations. It is <em>numerical</em> because it is not an algorithm which produces an exact solution, due to <a href="https://en.wikipedia.org/wiki/Numerical_error" rel="noreferrer">numerical errors</a> (e.g., round-off errors).</p>
  
  <p>GD is based on the idea that, if a (multi-variable) function <span class="math-container">$f(\mathbf{x})$</span> (where <span class="math-container">$\mathbf{x} \in \mathbb{R}^N$</span> is a variable representing the input of <span class="math-container">$f$</span>) is <em>defined</em> and <em>differentiable</em> in a "<a href="https://en.wikipedia.org/wiki/Neighbourhood_(mathematics)" rel="noreferrer">neighbourhood</a>" of a point <span class="math-container">$\mathbf{\theta} \in \mathbb{R}^N$</span>, which is a point (in the loose sense of the word "point") in the <a href="https://en.wikipedia.org/wiki/Domain_of_a_function" rel="noreferrer">domain</a> of <span class="math-container">$f$</span> (that is, <span class="math-container">$f(\mathbf{\theta})$</span> is defined and differentiable), then <span class="math-container">$f$</span> <strong>decreases fastest</strong> if one goes from <span class="math-container">$\mathbf{\theta}$</span> in the "direction" of the <strong>negative gradient</strong> of <span class="math-container">$f$</span> at <span class="math-container">$\mathbf{\theta}$</span>, that is, <span class="math-container">$-\nabla f(\mathbf {\theta})$</span>. (Do not be scared of the symbol <span class="math-container">$\theta$</span>, we could have denoted it by <span class="math-container">$\mathbf{a}$</span>!). Let us break this dense definition down!</p>
  
  <p>Note that here we assume that <span class="math-container">$f$</span> is a multi-variable function (that is, a function whose input is not just one real number, but it is a function of <span class="math-container">$N$</span> real numbers, which can be grouped in a vector, e.g. <span class="math-container">$\mathbf{\theta} \in \mathbb{R}^N$</span>), hence <span class="math-container">$\mathbf{x}$</span> and <span class="math-container">$\mathbf{\theta}$</span> are vectors of real numbers. To emphasize,  <span class="math-container">$\mathbf{\theta}$</span> is some specific (and "fixed") point in the domain of <span class="math-container">$f$</span>, whereas <span class="math-container">$\mathbf{x}$</span> is a variable that can take any of these specific points. </p>
  
  <p>The basic statement/assignment which is iterated in GD, which captures the definition above, is the following:</p>
  
  <p><span class="math-container">$$\mathbf{\theta}_{n+1} \gets \mathbf{\theta}_{n} - \gamma \nabla f(\mathbf{\theta} _{n}) \label{gd-assignment}\tag{1}$$</span></p>
  
  <p>where the <span class="math-container">$\gamma$</span> is a <a href="https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)" rel="noreferrer">hyper-parameter</a> which represents the "strength" of the step we take in the direction of the negative gradient (i.e., <span class="math-container">$-\nabla f(\mathbf{\theta} _{n})$</span>). Note the "minus" (i.e., <span class="math-container">$-$</span>) in front of <span class="math-container">$\nabla f(\mathbf{\theta}_{n})$</span>, hence the expression "in the direction of the <em>negative</em> gradient". Note also that here "direction" is implemented as a subtraction. The subscript <span class="math-container">$_n$</span> represents the current iteration number. </p>
  
  <p>As simple as it looks, the GD algorithm consists in the execution of the assignment \ref{gd-assignment} until some condition is met (e.g., <span class="math-container">${\theta}_{n+1}$</span> is "close enough" to <span class="math-container">${\theta}_{n}$</span>). In a nutshell, GD is an iterative algorithm which updates <span class="math-container">$\theta$</span> by iterating \ref{gd-assignment} multiple times.</p>
  
  <p>In the context of machine learning (ML), in particular supervised learning, we initially have an untrained model <span class="math-container">$\mathcal{M}$</span> (e.g., a neural network), that is, a model whose parameters (sometimes also called "weights") are e.g. randomly initialised. To train <span class="math-container">$\mathcal{M}$</span> (i.e., to find the most appropriate parameters of the model with respect to the training data), we first define a so-called "loss function" (which is differentiable), denoted by <span class="math-container">$\mathcal{L}$</span>, which represents the discrepancy between the current output of <span class="math-container">$\mathcal{M}$</span> and the expected output (with respect to the training data) of <span class="math-container">$\mathcal{M}$</span>. The process of <em>training a model</em> consists in minimising this loss function, that is, it consists in finding the minimum of <span class="math-container">$\mathcal{L}$</span>. </p>
  
  <p>To do this, as we have just seen above, we can use GD. If you consider the learning process the one of minimising this loss function <span class="math-container">$\mathcal{L}$</span>, then any algorithm which minimises <span class="math-container">$\mathcal{L}$</span> should be called a "learning algorithm". Hence GD should be called a "learning algorithm". </p>
  
  <p>Note that the parameters of the model <span class="math-container">$\mathcal{M}$</span>, which need to be found during the training process, are often denoted, in ML, by <span class="math-container">$\theta$</span>. So, the reason why I used the strange symbol <span class="math-container">$\theta$</span> in the assignment <span class="math-container">$\ref{gd-assignment}$</span> should now be clearer.</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Backpropagation" rel="noreferrer"><strong>Back-propagation</strong></a> (BP) is just an algorithm, <a href="http://people.idsia.ch/~juergen/who-invented-backpropagation.html" rel="noreferrer">proposed by Seppo Linnainmaa in his master's thesis</a>, to compute the derivative of a <em>differentiable</em> (composite) function, which can be represented as a graph.</p>
  
  <p>In ML, back-propagation is often used to compute <span class="math-container">$\nabla f(\mathbf{\theta}_{n})$</span> in the assignment \ref{gd-assignment}. So, BP is used as a sub-routine in the GD algorithm (or any other optimization algorithm) to find the parameters of the model <span class="math-container">$\mathcal{M}$</span>. </p>
  
  <p>Note that, in the context of ML, <span class="math-container">$\gamma$</span> in \ref{gd-assignment} is often called the "<em>learning</em> rate". There's a reason for this: GD is the actual learning algorithm. BP is just a sub-routine used to compute the gradient, but we could have used another sub-routine. We could say, in the context of ML (and, in particular, NNs), that the combination of GD and BP is the actual "learning algorithm", but, if by "learning" we mean "optimisation", then the actual "learning algorithm" is GD and <strong>not</strong> BP, because BP does 
  <em>not</em> optimise anything alone.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-15T13:51:51.137Z
Answers.LastActivityDate: 2018-11-15T13:51:51.137Z
Answers.UserDisplayName: nbro
Answers.Body: <p>You can run gradient descent without back propagation, in some cases:</p>
  
  <ul>
  <li><p>Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost function value.</p></li>
  <li><p>In "black box" gradient-based learning algorithms where you don't know how (or don't want to) calculate gradient analytically, so you choose to <em>measure</em> it. This is a slow process, similar to <a href="http://deeplearning.stanford.edu/tutorial/supervised/DebuggingGradientChecking/" rel="noreferrer">gradient checking</a> of neural networks, i.e. adjust each parameter in turn by a small <span class="math-container">$+\epsilon$</span>, measure your loss function, and divide difference by <span class="math-container">$\epsilon$</span> to get gradient estimate for that parameter.</p></li>
  </ul>
  
  <p>In those cases, you can calculate the gradient, and then use the optimisation function of your choice - e.g. SGD, Adam, Nesterov Momentum - to make changes to the learned parameters. The model will be changed and should learn its task based on the cost function.</p>
  
  <p>You can also run back propagation without gradient descent. In that case, the model will not change and nothing is learned.</p>
  
  <p>That makes the different flavours of gradient descent technically the "core" learning algorithm in my opinion, and back propagation the preferred method for calculating gradients in neural networks. These gradients are most often estimates of an unknown "true gradient" due to training in batches of small samples of training data.</p>
  
  <p>In practice, the two terms back propagation and gradient descent are rarely separated when discussing neural network training. So a lot of people will say that they train a network "using back propagation" or that they are using a gradient descent method without mentioning how they are calculating the gradients using back propagation (because that's pretty much a given). Or they might just say they are "using the Adam optimiser" which in the library they are using automatically runs an NN forward on a mini-batch of data, performs back propagation*, and then changes weight parameters based on the gradients, but heavily adjusted by previous gradient measurements.</p>
  
  <hr>
  
  <p>* With auto-differentiating frameworks such as TensorFlow, PyTorch etc, it is not really clear that the algorithms are performing "classic back propagation" - although they work using the same chain rule, they may well not perform calculations in the exact same groups that you learned when you studied neural networks and had to write the code for calculating gradients by hand. Instead of back propagating gradients layer-by-layer, they are propagated through an abstract computation graph that includes the learnable parameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T08:14:23.530Z
Answers.LastActivityDate: 2018-11-14T08:32:26.597Z
Answers.LastEditDate: 2018-11-14T08:32:26.597Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p><strong>Gradient descent</strong> is a very general optimization algorithm. <strong>Backpropagation</strong> is a special case of auto-differenciation combined with gradient descent. So backpropagation is a clever way to do gradient descent.</p>
  
  <p>The idea of Gradient descent is to calculate the gradient (derivative) and use that to descent on the error surface. In 3D, you could imagine 2 parameters (x,y) and the 3rd dimension (z) gives you the training error the model makes with those two parameters. You are at some (random) starting point in this error surface and make a step in downwards direction. Then you calculate the error again and do again a step in downwards direction. Note that I didn't write "downwards" but "in downwards direction" as your step might be too big.</p>
  
  <p>Backpropagation consists of two steps: The forward-pass and the backward-pass. In the forward pass, the output of the network for a given input is calculated. In the backward pass, the gradients are calculated with the chain rule.</p>
  
  <p>See <a href="http://cs231n.github.io/optimization-2/" rel="nofollow noreferrer">http://cs231n.github.io/optimization-2/</a> for a good reference for back propagation.</p>
  
  <p>Please note that machine learning is way broader than neural networks. For example, decision trees don't use gradient descent at all to train.</p>
  
  <blockquote>
    <p>What is the actual learning algorithm: back-propagation or gradient descent?</p>
  </blockquote>
  
  <p>Gradient descent is the learning algorithm. It is realized in backpropagation in the weight-update of the backward pass.</p>
  
  <h2>Notes and References</h2>
  
  <p>When I write "backpropagation algorithm", I refer to Tom Mitchells "Machine Learning", Chapter 4.5.2.</p>
  
  <p>To cite Tom Mitchells "Machine Learning":</p>
  
  <blockquote>
    <p>Algorithms such as BACKPROPAGATION use gradient descent to tune network parameters to best fit a training set of input-output pairs.</p>
    
    <p>[...] gradient descent provides the basis
    for the BACKPROPAGATION algorithm [...]</p>
    
    <p>[The section 4.5 MULTILAYER NETWORKS AND THE BACKPROPAGATION
    ALGORITHM] discusses how to learn such multilayer networks using a gradient descent algorithm similar to that discussed in the previous section. </p>
    
    <p>[...]</p>
    
    <p>As shown above, the BACKPROPAGATION algorithm implements a gradient descent search through the space of possible network weights</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-05T15:22:44.310Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105199/discussion-on-answer-by-martin-thoma-what-is-the-actual-learning-algorithm-back).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-11-14T06:34:42.887Z
Answers.LastActivityDate: 2018-11-15T06:14:35.640Z
Answers.LastEditDate: 2018-11-15T06:14:35.640Z
Answers.UserDisplayName: Martin Thoma
Body: <p>What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)?</p>
  
  <p>I am reading through chapter 8 of Parallel Distributed Processing hand book and the title of the chapter is "Learning internal representation by error propagation" by PDP research Group. 
  <a href="https://web.stanford.edu/class/psych209a/ReadingsByDate/02_06/PDPVolIChapter8.pdf" rel="noreferrer">https://web.stanford.edu/class/psych209a/ReadingsByDate/02_06/PDPVolIChapter8.pdf</a></p>
  
  <p>If there are no hidden units, no learning happens.
  If there are hidden units, they learn internal representation by propagating error back.
  Does this mean back propagation[delta rule] is the learning rule and gradient descent is an optimization algorithm used to optimize cost function?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-14T21:28:48.757Z
Comments.Text: See also the discussion here: https://stackoverflow.com/q/37953585/3924118.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-11-17T12:45:07.300Z
Comments.Text: "If there are no hidden units, no learning happens. If there are hidden units, they learn internal representation by propagating error back.", this is a property of the model, not of the learning algorithm.
Comments.UserDisplayName: nbro
Tags: <machine-learning><backpropagation><terminology><optimization>
Title: What is the actual learning algorithm: back-propagation or gradient descent?
UserDisplayName: Sreedhar Veluri
ThreadText: # What is the actual learning algorithm: back-propagation or gradient descent?
  
  (Asked by: username_0 on 2018-11-13)
  
  What is the actual learning algorithm: back-propagation or gradient descent (or, in general, the optimization algorithm)?
  
  I am reading through chapter 8 of Parallel Distributed Processing hand book and the title of the chapter is "Learning internal representation by error propagation" by PDP research Group.
  <https://web.stanford.edu/class/psych209a/ReadingsByDate/02_06/PDPVolIChapter8.pdf>
  
  If there are no hidden units, no learning happens.
  If there are hidden units, they learn internal representation by propagating error back.
  Does this mean back propagation[delta rule] is the learning rule and gradient descent is an optimization algorithm used to optimize cost function?
  
  **Tags:** machine-learning, backpropagation, terminology, optimization
  > **Comments:**
  > **username_1**: See also the discussion here: https://stackoverflow.com/q/37953585/3924118.  
  > **username_1**: "If there are no hidden units, no learning happens. If there are hidden units, they learn internal representation by propagating error back.", this is a property of the model, not of the learning algorithm.  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2018-11-14. Score: 7
  
  You can run gradient descent without back propagation, in some cases:
  
  * Simple structures such as linear or logistic regression, where the gradients can be calculated directly from the inputs and cost function value.
  * In "black box" gradient-based learning algorithms where you don't know how (or don't want to) calculate gradient analytically, so you choose to *measure* it. This is a slow process, similar to [gradient checking](http://deeplearning.stanford.edu/tutorial/supervised/DebuggingGradientChecking/) of neural networks, i.e. adjust each parameter in turn by a small $+\epsilon$, measure your loss function, and divide difference by $\epsilon$ to get gradient estimate for that parameter.
  
  In those cases, you can calculate the gradient, and then use the optimisation function of your choice - e.g. SGD, Adam, Nesterov Momentum - to make changes to the learned parameters. The model will be changed and should learn its task based on the cost function.
  
  You can also run back propagation without gradient descent. In that case, the model will not change and nothing is learned.
  
  That makes the different flavours of gradient descent technically the "core" learning algorithm in my opinion, and back propagation the preferred method for calculating gradients in neural networks. These gradients are most often estimates of an unknown "true gradient" due to training in batches of small samples of training data.
  
  In practice, the two terms back propagation and gradient descent are rarely separated when discussing neural network training. So a lot of people will say that they train a network "using back propagation" or that they are using a gradient descent method without mentioning how they are calculating the gradients using back propagation (because that's pretty much a given). Or they might just say they are "using the Adam optimiser" which in the library they are using automatically runs an NN forward on a mini-batch of data, performs back propagation\*, and then changes weight parameters based on the gradients, but heavily adjusted by previous gradient measurements.
  
  ---
  
  \* With auto-differentiating frameworks such as TensorFlow, PyTorch etc, it is not really clear that the algorithms are performing "classic back propagation" - although they work using the same chain rule, they may well not perform calculations in the exact same groups that you learned when you studied neural networks and had to write the code for calculating gradients by hand. Instead of back propagating gradients layer-by-layer, they are propagated through an abstract computation graph that includes the learnable parameters.
  
  
  ---
  
  ### Answer by username_1 on 2018-11-15. Score: 5
  
  [**Gradient descent**](https://en.wikipedia.org/wiki/Gradient_descent) (GD) is an optimisation algorithm, that is, it is used to find a (local) minimum of a [multi-variable](https://en.wikipedia.org/wiki/Function_of_several_real_variables) and [differentiable](https://en.wikipedia.org/wiki/Differentiable_function) function $f$. GD is an *iterative* and *numerical* optimisation algorithm. It is *iterative* because it proceeds in iterations. It is *numerical* because it is not an algorithm which produces an exact solution, due to [numerical errors](https://en.wikipedia.org/wiki/Numerical_error) (e.g., round-off errors).
  
  GD is based on the idea that, if a (multi-variable) function $f(\mathbf{x})$ (where $\mathbf{x} \in \mathbb{R}^N$ is a variable representing the input of $f$) is *defined* and *differentiable* in a "[neighbourhood](https://en.wikipedia.org/wiki/Neighbourhood_(mathematics))" of a point $\mathbf{\theta} \in \mathbb{R}^N$, which is a point (in the loose sense of the word "point") in the [domain](https://en.wikipedia.org/wiki/Domain_of_a_function) of $f$ (that is, $f(\mathbf{\theta})$ is defined and differentiable), then $f$ **decreases fastest** if one goes from $\mathbf{\theta}$ in the "direction" of the **negative gradient** of $f$ at $\mathbf{\theta}$, that is, $-\nabla f(\mathbf {\theta})$. (Do not be scared of the symbol $\theta$, we could have denoted it by $\mathbf{a}$!). Let us break this dense definition down!
  
  Note that here we assume that $f$ is a multi-variable function (that is, a function whose input is not just one real number, but it is a function of $N$ real numbers, which can be grouped in a vector, e.g. $\mathbf{\theta} \in \mathbb{R}^N$), hence $\mathbf{x}$ and $\mathbf{\theta}$ are vectors of real numbers. To emphasize, $\mathbf{\theta}$ is some specific (and "fixed") point in the domain of $f$, whereas $\mathbf{x}$ is a variable that can take any of these specific points.
  
  The basic statement/assignment which is iterated in GD, which captures the definition above, is the following:
  
  $$\mathbf{\theta}\_{n+1} \gets \mathbf{\theta}\_{n} - \gamma \nabla f(\mathbf{\theta} \_{n}) \label{gd-assignment}\tag{1}$$
  
  where the $\gamma$ is a [hyper-parameter](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)) which represents the "strength" of the step we take in the direction of the negative gradient (i.e., $-\nabla f(\mathbf{\theta} \_{n})$). Note the "minus" (i.e., $-$) in front of $\nabla f(\mathbf{\theta}\_{n})$, hence the expression "in the direction of the *negative* gradient". Note also that here "direction" is implemented as a subtraction. The subscript $\_n$ represents the current iteration number.
  
  As simple as it looks, the GD algorithm consists in the execution of the assignment \ref{gd-assignment} until some condition is met (e.g., ${\theta}\_{n+1}$ is "close enough" to ${\theta}\_{n}$). In a nutshell, GD is an iterative algorithm which updates $\theta$ by iterating \ref{gd-assignment} multiple times.
  
  In the context of machine learning (ML), in particular supervised learning, we initially have an untrained model $\mathcal{M}$ (e.g., a neural network), that is, a model whose parameters (sometimes also called "weights") are e.g. randomly initialised. To train $\mathcal{M}$ (i.e., to find the most appropriate parameters of the model with respect to the training data), we first define a so-called "loss function" (which is differentiable), denoted by $\mathcal{L}$, which represents the discrepancy between the current output of $\mathcal{M}$ and the expected output (with respect to the training data) of $\mathcal{M}$. The process of *training a model* consists in minimising this loss function, that is, it consists in finding the minimum of $\mathcal{L}$.
  
  To do this, as we have just seen above, we can use GD. If you consider the learning process the one of minimising this loss function $\mathcal{L}$, then any algorithm which minimises $\mathcal{L}$ should be called a "learning algorithm". Hence GD should be called a "learning algorithm".
  
  Note that the parameters of the model $\mathcal{M}$, which need to be found during the training process, are often denoted, in ML, by $\theta$. So, the reason why I used the strange symbol $\theta$ in the assignment $\ref{gd-assignment}$ should now be clearer.
  
  [**Back-propagation**](https://en.wikipedia.org/wiki/Backpropagation) (BP) is just an algorithm, [proposed by Seppo Linnainmaa in his master's thesis](http://people.idsia.ch/~juergen/who-invented-backpropagation.html), to compute the derivative of a *differentiable* (composite) function, which can be represented as a graph.
  
  In ML, back-propagation is often used to compute $\nabla f(\mathbf{\theta}\_{n})$ in the assignment \ref{gd-assignment}. So, BP is used as a sub-routine in the GD algorithm (or any other optimization algorithm) to find the parameters of the model $\mathcal{M}$.
  
  Note that, in the context of ML, $\gamma$ in \ref{gd-assignment} is often called the "*learning* rate". There's a reason for this: GD is the actual learning algorithm. BP is just a sub-routine used to compute the gradient, but we could have used another sub-routine. We could say, in the context of ML (and, in particular, NNs), that the combination of GD and BP is the actual "learning algorithm", but, if by "learning" we mean "optimisation", then the actual "learning algorithm" is GD and **not** BP, because BP does
  *not* optimise anything alone.
  
  
  ---
  
  ### Answer by username_3 on 2018-11-14. Score: 1
  
  **Gradient descent** is a very general optimization algorithm. **Backpropagation** is a special case of auto-differenciation combined with gradient descent. So backpropagation is a clever way to do gradient descent.
  
  The idea of Gradient descent is to calculate the gradient (derivative) and use that to descent on the error surface. In 3D, you could imagine 2 parameters (x,y) and the 3rd dimension (z) gives you the training error the model makes with those two parameters. You are at some (random) starting point in this error surface and make a step in downwards direction. Then you calculate the error again and do again a step in downwards direction. Note that I didn't write "downwards" but "in downwards direction" as your step might be too big.
  
  Backpropagation consists of two steps: The forward-pass and the backward-pass. In the forward pass, the output of the network for a given input is calculated. In the backward pass, the gradients are calculated with the chain rule.
  
  See <http://cs231n.github.io/optimization-2/> for a good reference for back propagation.
  
  Please note that machine learning is way broader than neural networks. For example, decision trees don't use gradient descent at all to train.
  
  > What is the actual learning algorithm: back-propagation or gradient descent?
  
  Gradient descent is the learning algorithm. It is realized in backpropagation in the weight-update of the backward pass.
  
  ## Notes and References
  
  When I write "backpropagation algorithm", I refer to Tom Mitchells "Machine Learning", Chapter 4.5.2.
  
  To cite Tom Mitchells "Machine Learning":
  
  > Algorithms such as BACKPROPAGATION use gradient descent to tune network parameters to best fit a training set of input-output pairs.
  >
  > [...] gradient descent provides the basis
  > for the BACKPROPAGATION algorithm [...]
  >
  > [The section 4.5 MULTILAYER NETWORKS AND THE BACKPROPAGATION
  > ALGORITHM] discusses how to learn such multilayer networks using a gradient descent algorithm similar to that discussed in the previous section.
  >
  > [...]
  >
  > As shown above, the BACKPROPAGATION algorithm implements a gradient descent search through the space of possible network weights
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/105199/discussion-on-answer-by-martin-thoma-what-is-the-actual-learning-algorithm-back).  
  
  ---
  
  