Answers.Body: <p>The concern you've raised touches on the method of training known as online learning or <a href="https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method" rel="nofollow noreferrer">stochastic gradient descent (SGD)</a>, where weights are updated after each individual training example. It's a common misunderstanding that, when using SGD even though weights are updated after each individual training example, the process involves iterating over the entire dataset multiple times. Each iteration is called an epoch. There's also a simple example in the cited reference.</p>
  <blockquote>
  <p>As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges.</p>
  </blockquote>
  <p>The network is exposed to each training example many times over multiple epochs. Each example contributes to the weight updates incrementally, ensuring the model learns from the entire dataset about which the usual empirical risk minimization (ERM) ML framework targets to minimize its average loss. The updates in SGD are noisy but more efficient than batch gradient descent using the entire datatset, which helps the model escape local minima and potentially find a better global minimum.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-14T20:27:56.203Z
Answers.Text: Awesome, thank you. It seems convergence is key here, and that makes sense.
Answers.UserDisplayName: thanks_in_advance
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T16:17:48.553Z
Answers.Text: @cinch Also the choice of the learning rate (or step size) is important: the LR is usually quite small (e.g., $3\cdot 10^{-4}$), so that a small gradient step yields the new weights, which, on average and assuming infinite data, converge to the optimal value. If the LR is large (say $1$) you may still overfit the last seen data sample (assuming SGD with batch size of 1.) because you're adapting too much
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-19T17:20:36.837Z
Answers.Text: @LucaAnzalone thanks for your comment and I totally agree as I also emphasized LR role in the last quoted sentence above, and incidentally I also emphasized the importance of the small LR and its schedule in my recent [answer](https://ai.stackexchange.com/questions/35648/at-which-point-does-the-momentum-based-gd-helps-really-in-this-figure/46171#46171) to another post, but seems no one pays any attention.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-14T20:23:22.103Z
Answers.LastActivityDate: 2024-07-14T20:23:22.103Z
Answers.UserDisplayName: cinch
Body: <p>Here's a BackProp Algo definition from <a href="https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/" rel="nofollow noreferrer">here</a>:</p>
  <blockquote>
  <p>Initially all the edge weights are randomly assigned. For every input in the training dataset, the ANN is activated and its output is observed. This output is compared with the desired output that we already know, and the error is “propagated” back to the previous layer. This error is noted and the weights are “adjusted” accordingly. This process is repeated until the output error is below a predetermined threshold.</p>
  </blockquote>
  <p>Something I'm not understanding here: <em><strong>If inputs are fed one by one, and the weights are adjusted for each input, won't the NN essentially be trained for the last input?</strong></em></p>
  <p>Please clarify. Thank you.</p>
  
Tags: <backpropagation>
Title: How does BackProp avoid bias for the last input used for training?
UserDisplayName: thanks_in_advance
ThreadText: # How does BackProp avoid bias for the last input used for training?
  
  (Asked by: username_0 on 2024-07-14)
  
  Here's a BackProp Algo definition from [here](https://ujjwalkarn.me/2016/08/09/quick-intro-neural-networks/):
  
  > Initially all the edge weights are randomly assigned. For every input in the training dataset, the ANN is activated and its output is observed. This output is compared with the desired output that we already know, and the error is “propagated” back to the previous layer. This error is noted and the weights are “adjusted” accordingly. This process is repeated until the output error is below a predetermined threshold.
  
  Something I'm not understanding here: ***If inputs are fed one by one, and the weights are adjusted for each input, won't the NN essentially be trained for the last input?***
  
  Please clarify. Thank you.
  
  **Tags:** backpropagation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-14. Score: 3 | Accepted answer
  
  The concern you've raised touches on the method of training known as online learning or [stochastic gradient descent (SGD)](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Iterative_method), where weights are updated after each individual training example. It's a common misunderstanding that, when using SGD even though weights are updated after each individual training example, the process involves iterating over the entire dataset multiple times. Each iteration is called an epoch. There's also a simple example in the cited reference.
  
  > As the algorithm sweeps through the training set, it performs the above update for each training sample. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles. Typical implementations may use an adaptive learning rate so that the algorithm converges.
  
  The network is exposed to each training example many times over multiple epochs. Each example contributes to the weight updates incrementally, ensuring the model learns from the entire dataset about which the usual empirical risk minimization (ERM) ML framework targets to minimize its average loss. The updates in SGD are noisy but more efficient than batch gradient descent using the entire datatset, which helps the model escape local minima and potentially find a better global minimum.
  
  > **Comments:**
  > **username_0**: Awesome, thank you. It seems convergence is key here, and that makes sense.  
  > **username_2**: @cinch Also the choice of the learning rate (or step size) is important: the LR is usually quite small (e.g., $3\cdot 10^{-4}$), so that a small gradient step yields the new weights, which, on average and assuming infinite data, converge to the optimal value. If the LR is large (say $1$) you may still overfit the last seen data sample (assuming SGD with batch size of 1.) because you're adapting too much  
  > **username_1**: @LucaAnzalone thanks for your comment and I totally agree as I also emphasized LR role in the last quoted sentence above, and incidentally I also emphasized the importance of the small LR and its schedule in my recent [answer](https://ai.stackexchange.com/questions/35648/at-which-point-does-the-momentum-based-gd-helps-really-in-this-figure/46171#46171) to another post, but seems no one pays any attention.  
  
  ---
  
  