Answers.Body: <p>I would say just don't go for using regularization at first, try with lower <code>learning rate =  0.0001</code> see the behavior. Try to post entire architecture of your model so that one can better answer your problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-03T07:04:49.543Z
Answers.Text: Yeah. I had posted the code. Tried Xavier initialisation. Decreased cost down to 0.0000075 and it converges fine. So I took it down.
Answers.UserDisplayName: pranav
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-03T04:38:53.470Z
Answers.LastActivityDate: 2018-08-03T04:38:53.470Z
Answers.UserDisplayName: Patel Sunil
Answers.Body: <p>I figured out the problem after a bit of trial and error. This <a href="http://thelaziestprogrammer.com/sharrington/math-of-machine-learning/gradient-descent-learning-rate-too-high?utm_medium=email&amp;utm_source=other&amp;utm_campaign=opencourse.discourse.neural-networks-deep-learning~opencourse.discourse.W_mOXCrdEeeNPQ68_4aPpA.2GpetZVyEei1PwqN7iH8Jg~FwF3ipXIEeir3xJNYGdMZA" rel="nofollow noreferrer">article</a> may help too</p>
  
  <p>First i set about pruning the dataset and removed outliers. Then i initialised the weights to better values using xavier initialisation. 
  These made the model slightly better but still the problem occured.</p>
  
  <p>I then set the learning rate down to 0.000000075 and it converged after about 10000 iterations. I guess it was overshooting the minimum and reaching a point on the other side of the minimum that was farther than the previous point. This resulted in increased magnitude of gradient in the opposite direction and the cost explosively went up to the order of billions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-03T19:13:39.803Z
Answers.Text: @DuttA I would hazard a guess that the OP has not normalised the inputs.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-08-03T13:46:48.853Z
Answers.LastActivityDate: 2018-08-03T14:06:44.907Z
Answers.LastEditDate: 2018-08-03T14:06:44.907Z
Answers.UserDisplayName: pranav
Body: <p>In doing a project using neural networks with an input layer, 4 hidden layers and an output layer ,I used mini batch gradient descent. I noticed that the randomly initialised weights seemed to do a good performance and gave a low error. As the model started training after about 200 iterations there was large jump in error and then it came down slowly from there. I have also noticed that sometimes the cost just increases over a set of consecutive iterations.
  Can anyone explain why these happen? It is not like there are outliers or a new distribution as every iteration exposes it to the entire dataset. 
  I used learning rate 0.01 and regularisation parameter 10. I also tried regularisation parameter 5 and also 1.
  And by the cost  I mean, the sum of squared errors of all minibatches/2m plus regularisation term error.
  Further if this happens and my cost after the say 10000th iteration is more than my cost when I initialised with random weights (lol) can I just take the initial value? As those weights seem to be doing better.</p>
  
  <p>The large jumps are the most puzzling. </p>
  
  <p>This is the code</p>
  
  <p>Any help would be greatly appreciated. Thanks</p>
  
Tags: <neural-networks><gradient-descent>
Title: Behaviour of cost
UserDisplayName: pranav
ThreadText: # Behaviour of cost
  
  (Asked by: username_0 on 2018-07-31)
  
  In doing a project using neural networks with an input layer, 4 hidden layers and an output layer ,I used mini batch gradient descent. I noticed that the randomly initialised weights seemed to do a good performance and gave a low error. As the model started training after about 200 iterations there was large jump in error and then it came down slowly from there. I have also noticed that sometimes the cost just increases over a set of consecutive iterations.
  Can anyone explain why these happen? It is not like there are outliers or a new distribution as every iteration exposes it to the entire dataset.
  I used learning rate 0.01 and regularisation parameter 10. I also tried regularisation parameter 5 and also 1.
  And by the cost I mean, the sum of squared errors of all minibatches/2m plus regularisation term error.
  Further if this happens and my cost after the say 10000th iteration is more than my cost when I initialised with random weights (lol) can I just take the initial value? As those weights seem to be doing better.
  
  The large jumps are the most puzzling.
  
  This is the code
  
  Any help would be greatly appreciated. Thanks
  
  **Tags:** neural-networks, gradient-descent
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-08-03. Score: 1
  
  I would say just don't go for using regularization at first, try with lower `learning rate = 0.0001` see the behavior. Try to post entire architecture of your model so that one can better answer your problem.
  
  > **Comments:**
  > **username_0**: Yeah. I had posted the code. Tried Xavier initialisation. Decreased cost down to 0.0000075 and it converges fine. So I took it down.  
  
  ---
  
  ### Answer by username_0 on 2018-08-03. Score: 1
  
  I figured out the problem after a bit of trial and error. This [article](http://thelaziestprogrammer.com/sharrington/math-of-machine-learning/gradient-descent-learning-rate-too-high?utm_medium=email&utm_source=other&utm_campaign=opencourse.discourse.neural-networks-deep-learning~opencourse.discourse.W_mOXCrdEeeNPQ68_4aPpA.2GpetZVyEei1PwqN7iH8Jg~FwF3ipXIEeir3xJNYGdMZA) may help too
  
  First i set about pruning the dataset and removed outliers. Then i initialised the weights to better values using xavier initialisation.
  These made the model slightly better but still the problem occured.
  
  I then set the learning rate down to 0.000000075 and it converged after about 10000 iterations. I guess it was overshooting the minimum and reaching a point on the other side of the minimum that was farther than the previous point. This resulted in increased magnitude of gradient in the opposite direction and the cost explosively went up to the order of billions.
  
  > **Comments:**
  > **username_2**: @DuttA I would hazard a guess that the OP has not normalised the inputs.  
  
  ---
  
  