Answers.Body: <p>Yes. Not only that, but error is highly noisy, prone to big spikes and sometimes quite long period of increase before decrease again or stabilize. Often it's even impossible to understand error plot without passing it through smoothing filter, so noisy it is. Specific depend on the problem of cause. It's not only for SGD but for any optimizer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-07-10T05:33:59.047Z
Answers.LastActivityDate: 2019-07-10T05:33:59.047Z
Answers.UserDisplayName: mirror2image
Body: <p>As simple as that. Is there any scenario where the error might <em>increase</em>, if only by a tiny amount, when using SGD (no momentum)?</p>
  
Tags: <neural-networks><gradient-descent><objective-functions>
Title: Is it possible with stochastic gradient descent for the error to increase?
UserDisplayName: Recessive
ThreadText: # Is it possible with stochastic gradient descent for the error to increase?
  
  (Asked by: username_0 on 2019-07-10)
  
  As simple as that. Is there any scenario where the error might *increase*, if only by a tiny amount, when using SGD (no momentum)?
  
  **Tags:** neural-networks, gradient-descent, objective-functions
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-07-10. Score: 3 | Accepted answer
  
  Yes. Not only that, but error is highly noisy, prone to big spikes and sometimes quite long period of increase before decrease again or stabilize. Often it's even impossible to understand error plot without passing it through smoothing filter, so noisy it is. Specific depend on the problem of cause. It's not only for SGD but for any optimizer.
  
  
  ---
  
  