Answers.Body: <p>You are using two optimisers here: Stochastic Gradient Descent (SGD) and Adam (which is a more complex variant of SGD).</p>
  
  <p>So the "Stochastic" part means that it's random.</p>
  
  <p>The stochastic gradient descent works by taking a smaller <em>random</em> part of the training data, called "mini batch", and back propagates (trains) on this. Doing this until the entire dataset is processed once is often called one epoch*. </p>
  
  <p>This is how gradient descent works in a nutshell: Imagine you're going down a U-shaped hill. You're pretty far down in the U-shape, and you want to go further down by jumping. You figure out what direction is "down" for you: and then you jump. But darn it: you jumped too far and you ended up further up on the other side of the U!</p>
  
  <p>That is just a simple example. You are probably working in WAY bigger dimensions, which complicates this analogy a bit.</p>
  
  <p>Anyway this results in the effect that the loss might go up from time to time when you train another epoch. If you are training a lot of epochs and the loss keeps going up, you should check the learning rate (which basically decides how big a "jump" is).</p>
  
  <p>Hope it helps :)</p>
  
  <hr>
  
  <p>*: There are other ways of defining an epoch, but it all goes in variants of this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-04-10T20:04:24.937Z
Answers.LastActivityDate: 2018-05-31T22:09:15.300Z
Answers.LastEditDate: 2018-05-31T22:09:15.300Z
Answers.UserDisplayName: Andreas Storvik Strauman
Body: <p>I am using Keras to train different NN. I would like to know why if I increment the epochs in 1, the result until the new epoch is not the same. I am using shuffle=False, and np.random.seed(2017), and I have check that if I repeat with the same number of epochs, the result is the same, so not random initialization is working.</p>
  
  <p>Here I attach the picture of the resulting training with 2 epochs:</p>
  
  <p><a href="https://i.sstatic.net/DtLWl.png" rel="noreferrer"><img src="https://i.sstatic.net/DtLWl.png" alt=""></a>
  And here I attach the picture of the resulting training with 3 epochs:</p>
  
  <p><a href="https://i.sstatic.net/auDvm.png" rel="noreferrer"><img src="https://i.sstatic.net/auDvm.png" alt="enter image description here"></a></p>
  
  <p>Also, I would like to know why the training time is not (3/2) and how is it possible that some of them have less accuracy with one more epoch.</p>
  
  <p>Thanks a lot!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-06-01T08:45:39.563Z
Comments.Text: This question should be migrated to: https://datascience.stackexchange.com/.
Comments.UserDisplayName: JahKnows
Tags: <neural-networks><training><optimization><keras>
Title: Why does 'loss' change depending on the number of epochs chosen?
UserDisplayName: Pablo Ruiz Ruiz
ThreadText: # Why does 'loss' change depending on the number of epochs chosen?
  
  (Asked by: username_0 on 2017-12-07)
  
  I am using Keras to train different NN. I would like to know why if I increment the epochs in 1, the result until the new epoch is not the same. I am using shuffle=False, and np.random.seed(2017), and I have check that if I repeat with the same number of epochs, the result is the same, so not random initialization is working.
  
  Here I attach the picture of the resulting training with 2 epochs:
  
  [![](https://i.sstatic.net/DtLWl.png)](https://i.sstatic.net/DtLWl.png)
  And here I attach the picture of the resulting training with 3 epochs:
  
  [![enter image description here](https://i.sstatic.net/auDvm.png)](https://i.sstatic.net/auDvm.png)
  
  Also, I would like to know why the training time is not (3/2) and how is it possible that some of them have less accuracy with one more epoch.
  
  Thanks a lot!
  
  **Tags:** neural-networks, training, optimization, keras
  
  > **Comments:**
  > **username_1**: This question should be migrated to: https://datascience.stackexchange.com/.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-04-10. Score: 1
  
  You are using two optimisers here: Stochastic Gradient Descent (SGD) and Adam (which is a more complex variant of SGD).
  
  So the "Stochastic" part means that it's random.
  
  The stochastic gradient descent works by taking a smaller *random* part of the training data, called "mini batch", and back propagates (trains) on this. Doing this until the entire dataset is processed once is often called one epoch\*.
  
  This is how gradient descent works in a nutshell: Imagine you're going down a U-shaped hill. You're pretty far down in the U-shape, and you want to go further down by jumping. You figure out what direction is "down" for you: and then you jump. But darn it: you jumped too far and you ended up further up on the other side of the U!
  
  That is just a simple example. You are probably working in WAY bigger dimensions, which complicates this analogy a bit.
  
  Anyway this results in the effect that the loss might go up from time to time when you train another epoch. If you are training a lot of epochs and the loss keeps going up, you should check the learning rate (which basically decides how big a "jump" is).
  
  Hope it helps :)
  
  ---
  
  \*: There are other ways of defining an epoch, but it all goes in variants of this.
  
  
  ---
  
  