Answers.Body: <p>Both are feasible.</p>
  <p>A generalization is to apply the modification of the weights after the presentation of <span class="math-container">$n$</span> examples. If <span class="math-container">$n=1$</span>, this is <strong>online</strong> training. If <span class="math-container">$n=60$</span> (<span class="math-container">$60$</span>, in your case, is the size of the dataset), this is <strong>batch</strong> training. In other cases, this is <strong>mini-batch</strong> training.</p>
  <p>The main difference is not the computation complexity of the algorithm, but the theoretical speed of convergence to an &quot;optimal&quot; set of weights.</p>
  <p>It is now generally admitted that <em>online training</em> is faster. Online training performs a stochastic approximation of the gradient descent method. The true gradient of the cost function over the whole training set is approximated by a gradient at <strong>each</strong> presentation of a training example.</p>
  <p>Intuitively, with <em>batch learning</em> the weights are modified by an <strong>average</strong> of the gradient over the whole training set and averaging destroys information.</p>
  <p>While with stochastic (or mini-batch learning), each example has his word to say. One after the other.</p>
  <p>This has also been discussed <a href="https://stats.stackexchange.com/q/49528/82135">here</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-10T18:33:16.397Z
Answers.Text: You say "It is now generally admitted that online training is faster", but I am not sure. Faster in terms of convergence? Can you provide a reference (i.e. a reliable paper or book) that supports that claim?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-06-15T19:00:33.540Z
Answers.LastActivityDate: 2020-12-10T18:29:45.753Z
Answers.LastEditDate: 2020-12-10T18:29:45.753Z
Answers.UserDisplayName: jcm69
Body: <p>I am a machine learning newbie. I am trying to understand the back-propagation algorithm. I have a training dataset of 60 instances/records.</p>
  <p>What is the correct order of the process? This one?</p>
  <ul>
  <li>Forward pass of the first instance. Calculate the error.</li>
  <li>Weight update using backpropagation.</li>
  <li>Forward pass of the second instance. Calculate the error.</li>
  <li>Weight update using backpropagation. And so on...</li>
  </ul>
  <p>Or</p>
  <ul>
  <li>Forward pass of all instances one by one. Keep track of the error as a vector.</li>
  <li>Weight update using backpropagation.</li>
  </ul>
  <p>This video <a href="https://www.youtube.com/watch?v=OwdjRYUPngE" rel="nofollow noreferrer">https://www.youtube.com/watch?v=OwdjRYUPngE</a> is similar to the second process. Is it correct?</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-06-15T14:17:09.380Z
Comments.Text: I think the video you watched is an example of _batch training_. Meaning that you activate an x amount of samples from your set, and you calculate the error over those samples and then change the weights. So you change the weights every x samples. However, if you are a newbie, you should first focus on _online learning_: activating 1 sample, then backpropagating and repeating that for every sample in the set.
Comments.UserDisplayName: Thomas Wagenaar
Tags: <neural-networks><deep-learning><backpropagation><stochastic-gradient-descent><mini-batch-gradient-descent>
Title: What is the order of execution of steps in back-propagation algorithm in a neural network?
UserDisplayName: gokul
ThreadText: # What is the order of execution of steps in back-propagation algorithm in a neural network?
  
  (Asked by: username_0 on 2017-06-15)
  
  I am a machine learning newbie. I am trying to understand the back-propagation algorithm. I have a training dataset of 60 instances/records.
  
  What is the correct order of the process? This one?
  
  * Forward pass of the first instance. Calculate the error.
  * Weight update using backpropagation.
  * Forward pass of the second instance. Calculate the error.
  * Weight update using backpropagation. And so on...
  
  Or
  
  * Forward pass of all instances one by one. Keep track of the error as a vector.
  * Weight update using backpropagation.
  
  This video <https://www.youtube.com/watch?v=OwdjRYUPngE> is similar to the second process. Is it correct?
  
  **Tags:** neural-networks, deep-learning, backpropagation, stochastic-gradient-descent, mini-batch-gradient-descent
  > **Comments:**
  > **username_1**: I think the video you watched is an example of _batch training_. Meaning that you activate an x amount of samples from your set, and you calculate the error over those samples and then change the weights. So you change the weights every x samples. However, if you are a newbie, you should first focus on _online learning_: activating 1 sample, then backpropagating and repeating that for every sample in the set.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2017-06-15. Score: 0 | Accepted answer
  
  Both are feasible.
  
  A generalization is to apply the modification of the weights after the presentation of $n$ examples. If $n=1$, this is **online** training. If $n=60$ ($60$, in your case, is the size of the dataset), this is **batch** training. In other cases, this is **mini-batch** training.
  
  The main difference is not the computation complexity of the algorithm, but the theoretical speed of convergence to an "optimal" set of weights.
  
  It is now generally admitted that *online training* is faster. Online training performs a stochastic approximation of the gradient descent method. The true gradient of the cost function over the whole training set is approximated by a gradient at **each** presentation of a training example.
  
  Intuitively, with *batch learning* the weights are modified by an **average** of the gradient over the whole training set and averaging destroys information.
  
  While with stochastic (or mini-batch learning), each example has his word to say. One after the other.
  
  This has also been discussed [here](https://stats.stackexchange.com/q/49528/82135).
  
  > **Comments:**
  > **username_3**: You say "It is now generally admitted that online training is faster", but I am not sure. Faster in terms of convergence? Can you provide a reference (i.e. a reliable paper or book) that supports that claim?  
  
  ---
  
  