Answers.Body: <p>The two examples present essentially the same operation:</p>
  
  <ul>
  <li><p>In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output.</p></li>
  <li><p>Both examples use the logistic function for node activation (the derivative of the logistic function <span class="math-container">$s$</span> is <span class="math-container">$s(1 - s)$</span>. This derivative is obviously very easy to compute, and this is part of the reason why it was so widely used (these days the <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">ReLU</a> activation function is more popular, especially with convolutional networks).</p></li>
  <li><p>The first method also uses <a href="https://www.quora.com/What-does-momentum-mean-in-neural-networks" rel="nofollow noreferrer">momentum</a>.</p></li>
  </ul>
  
  <p>The main difference I can see is that in the first case backpropagation is iterative while in the second example it is performed in batch mode.</p>
  
  <p>The last video in the series by Welch Labs introduces a quasi-Newtonian method which offers the advantage of finding the minimum of the cost function by computing the <a href="https://en.wikipedia.org/wiki/Hessian_matrix" rel="nofollow noreferrer">Hessian</a> (matrix of second-order derivatives of the error with respect to the weights). However, this feels like comparing apples and oranges - the vanilla gradient descent does not use second-order information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-06-29T02:54:42.017Z
Answers.LastActivityDate: 2020-05-23T13:00:58.127Z
Answers.LastEditDate: 2020-05-23T13:00:58.127Z
Answers.UserDisplayName: cantordust
Body: <p>Just for fun, I am trying to develop a neural network.</p>
  
  <p>Now, for backpropagation I saw two techniques.</p>
  
  <p>The first one is used <a href="http://courses.cs.washington.edu/courses/cse599/01wi/admin/Assignments/bpn.html" rel="nofollow noreferrer">here</a> and in many other places too.</p>
  
  <p>What it does is:</p>
  
  <ul>
  <li>It computes the error for each output neuron.</li>
  <li>It backpropagates it into the network (calculating an error for each inner neuron).</li>
  <li><p>It updates the weights with the formula: <span class="math-container">$\Delta w_{l, m, n}=k \cdot E_{l+1, n} \cdot N_{l, m}$</span>, where </p>
  
  <ul>
  <li><span class="math-container">$\Delta w_{l, m, n}$</span> is the change in weight, </li>
  <li><span class="math-container">$k$</span> is the learning speed, </li>
  <li><span class="math-container">$E_{l+1, n}$</span> is the error of the neuron receiving the input from the synapse, and </li>
  <li><span class="math-container">$ N_{l, m}$</span> is the output sent on the synapse.</li>
  </ul></li>
  <li><p>It repeats for each entry of the dataset, as many times as required.</p></li>
  </ul>
  
  <p>However, the neural network proposed in <a href="https://www.youtube.com/watch?v=bxe2T-V8XRs&amp;list=PL77aoaxdgEVDrHoFOMKTjDdsa0p9iVtsR" rel="nofollow noreferrer">this tutorial</a> (also available on GitHub) uses a different technique:</p>
  
  <ul>
  <li>It uses an error function (the other method does have an error function, but it does not use it for training).</li>
  <li>It has another function which can compute the final error starting from the weights.</li>
  <li>It minimizes that function (through gradient descent).</li>
  </ul>
  
  <p>Now, which method should be used?</p>
  
  <p>I think the first one is the most used one (because I saw different examples using it), but does it work as well?</p>
  
  <p>In particular, I don't know:</p>
  
  <ul>
  <li>Isn't it more subject to local minimums (since it doesn't use quadratic functions)?</li>
  <li>Since the variation of each weight is influenced by the output value of its output neuron, don't entries of the dataset which just happen to produce higher values in the neurons (not just the output ones) influence the weights more than other entries?</li>
  </ul>
  
  <p>Now, I do prefer the first technique, because I find it simpler to implement and easier to think about.</p>
  
  <p>Though, if it does have the problems I mentioned (which I hope it doesn't), is there any actual reason to use it over the second method?</p>
  
Tags: <neural-networks><machine-learning><comparison><backpropagation>
Title: Are these two versions of back-propagation equivalent?
UserDisplayName: Aspie96
ThreadText: # Are these two versions of back-propagation equivalent?
  
  (Asked by: username_0 on 2016-09-04)
  
  Just for fun, I am trying to develop a neural network.
  
  Now, for backpropagation I saw two techniques.
  
  The first one is used [here](http://courses.cs.washington.edu/courses/cse599/01wi/admin/Assignments/bpn.html) and in many other places too.
  
  What it does is:
  
  * It computes the error for each output neuron.
  * It backpropagates it into the network (calculating an error for each inner neuron).
  * It updates the weights with the formula: $\Delta w\_{l, m, n}=k \cdot E\_{l+1, n} \cdot N\_{l, m}$, where
  
    + $\Delta w\_{l, m, n}$ is the change in weight,
    + $k$ is the learning speed,
    + $E\_{l+1, n}$ is the error of the neuron receiving the input from the synapse, and
    + $ N\_{l, m}$ is the output sent on the synapse.
  * It repeats for each entry of the dataset, as many times as required.
  
  However, the neural network proposed in [this tutorial](https://www.youtube.com/watch?v=bxe2T-V8XRs&list=PL77aoaxdgEVDrHoFOMKTjDdsa0p9iVtsR) (also available on GitHub) uses a different technique:
  
  * It uses an error function (the other method does have an error function, but it does not use it for training).
  * It has another function which can compute the final error starting from the weights.
  * It minimizes that function (through gradient descent).
  
  Now, which method should be used?
  
  I think the first one is the most used one (because I saw different examples using it), but does it work as well?
  
  In particular, I don't know:
  
  * Isn't it more subject to local minimums (since it doesn't use quadratic functions)?
  * Since the variation of each weight is influenced by the output value of its output neuron, don't entries of the dataset which just happen to produce higher values in the neurons (not just the output ones) influence the weights more than other entries?
  
  Now, I do prefer the first technique, because I find it simpler to implement and easier to think about.
  
  Though, if it does have the problems I mentioned (which I hope it doesn't), is there any actual reason to use it over the second method?
  
  **Tags:** neural-networks, machine-learning, comparison, backpropagation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-06-29. Score: 5
  
  The two examples present essentially the same operation:
  
  * In both cases, the network is trained with gradient descent using the backpropagated squared error computed at the output.
  * Both examples use the logistic function for node activation (the derivative of the logistic function $s$ is $s(1 - s)$. This derivative is obviously very easy to compute, and this is part of the reason why it was so widely used (these days the [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) activation function is more popular, especially with convolutional networks).
  * The first method also uses [momentum](https://www.quora.com/What-does-momentum-mean-in-neural-networks).
  
  The main difference I can see is that in the first case backpropagation is iterative while in the second example it is performed in batch mode.
  
  The last video in the series by Welch Labs introduces a quasi-Newtonian method which offers the advantage of finding the minimum of the cost function by computing the [Hessian](https://en.wikipedia.org/wiki/Hessian_matrix) (matrix of second-order derivatives of the error with respect to the weights). However, this feels like comparing apples and oranges - the vanilla gradient descent does not use second-order information.
  
  
  ---
  
  