Answers.Body: <blockquote>
  <p>What might be causing these, and how to eliminate (or greatly reduce) them?</p>
  </blockquote>
  <p>It is difficult to be sure just from the graph, but I note you are using the Adam optimiser. It shares with a few other optimisers (most notably RMSProp) that it divides current gradients by a rolling mean of recent gradients to set step sizes. This can cause some minor instability when gradients get close to zero for a long while before growing again. This might happen at a saddle point where only some fraction of the network parameters are critical to results for a few iterations before hitting some other direction of slope where changing the &quot;quiet&quot; parameters becomes important again. This is more likely to occur as loss values become small, and close to perfect convergence.</p>
  <p>There are a couple of hyperparameters in Adam that may reduce the effect:</p>
  <ul>
  <li><p>Reduce learning rate. The default learning rate in Adam is often set to 0.001, but you will find a lot of researchers will reduce that to magnitudes around 0.0001 or 0.00001</p>
  </li>
  <li><p>Increase epsilon. This is effectively the minimum rolling average gradient. The default is <code>1e-7</code> but it sometimes needs to be increased significantly, it depends on the loss surface. The <a href="https://keras.io/api/optimizers/adam/" rel="nofollow noreferrer">official documentation suggests</a> that it may even be useful to increase it up to <code>1.0</code></p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-13T20:05:06.260Z
Answers.LastActivityDate: 2021-03-13T20:10:53.203Z
Answers.LastEditDate: 2021-03-13T20:10:53.203Z
Answers.UserDisplayName: Neil Slater
Body: <p>Performing a prediction of a continuous y target using Keras, the simple structure of the code revolves around;</p>
  <pre><code>model = Sequential()  
  model.add(Dense(200, input_dim=15, activation= &quot;relu&quot;))  
  model.add(Dense(750, activation= &quot;relu&quot;))  
  model.add(Dense(500, activation= &quot;relu&quot;))  
  model.add(Dense(750, activation= &quot;relu&quot;))  
  model.add(Dense(500, activation= &quot;relu&quot;))  
  model.add(Dense(200, activation= &quot;relu&quot;))  
  model.add(Dense(100, activation= &quot;relu&quot;))  
  model.add(Dense(50, activation= &quot;relu&quot;))  
  model.add(Dense(1)) 
  
  model.compile(loss= 'mse' , optimizer='adam', metrics=['mse','mae'])  
  history=model.fit(X_train, y_train, batch_size=50,  epochs=150,  
                    verbose=1, validation_split=0.2)
  </code></pre>
  <p>This has resulted in the following metric chart;</p>
  <p><a href="https://i.sstatic.net/9YvjK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9YvjK.png" alt="enter image description here" /></a></p>
  <p><a href="https://i.sstatic.net/d6ygU.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/d6ygU.png" alt="enter image description here" /></a></p>
  <p>What might be causing these, and how to eliminate (or greatly reduce) them?</p>
  <p>UPDATE: Just reduced the learning rate to 0.0001 per Neil Slater's suggestion, and the loss curve may have had the spikes reduced, though the scale of the graph has changed. The training loss has increased from 0.00007 to 0.00037, the validation loss from 0.0014 to 0.002, and the prediction error increased from 0.037 to 0.046.</p>
  <p><a href="https://i.sstatic.net/Jtmxf.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Jtmxf.png" alt="enter image description here" /></a></p>
  <p>I then changed epsilon from it's value of 1e-07 to 0.1 and increased the epochs from 150 to 500. The validation loss increased to 0.0082 and the prediction error increased to 0.093, with the corresponding model loss shown below.</p>
  <p><a href="https://i.sstatic.net/gDJOC.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/gDJOC.png" alt="enter image description here" /></a></p>
  <p>While not an overall improvement at either step, this did remove the spikes as I requested, hence Neil's advice gives me additional considerations to explore and measure within the Adam optimizer (along with other optimizers), so I consider this to have been an important learning experience. One such exploration uncovered this <a href="https://ruder.io/optimizing-gradient-descent/index.html" rel="nofollow noreferrer">more detailed explanation of optimizers</a> than I had been exposed to before, as well as this <a href="https://vimeo.com/352686490" rel="nofollow noreferrer">3D visualization</a> of loss topologies and the effect differing optimizers and parameters have on finding the optimal minima (keep a sharp eye on the options being chosen in the upper right corner).</p>
  
Tags: <python><keras><mean-squared-error>
Title: "Porpoising" in latter stages of validation loss and MSE charts in Keras
UserDisplayName: Will
ThreadText: # "Porpoising" in latter stages of validation loss and MSE charts in Keras
  
  (Asked by: username_0 on 2021-03-13)
  
  Performing a prediction of a continuous y target using Keras, the simple structure of the code revolves around;
  model = Sequential()  
  model.add(Dense(200, input_dim=15, activation= "relu"))  
  model.add(Dense(750, activation= "relu"))  
  model.add(Dense(500, activation= "relu"))  
  model.add(Dense(750, activation= "relu"))  
  model.add(Dense(500, activation= "relu"))  
  model.add(Dense(200, activation= "relu"))  
  model.add(Dense(100, activation= "relu"))  
  model.add(Dense(50, activation= "relu"))  
  model.add(Dense(1)) 
  
  model.compile(loss= 'mse' , optimizer='adam', metrics=['mse','mae'])  
  history=model.fit(X_train, y_train, batch_size=50,  epochs=150,  
                    verbose=1, validation_split=0.2)
  
  This has resulted in the following metric chart;
  
  
  What might be causing these, and how to eliminate (or greatly reduce) them?
  UPDATE: Just reduced the learning rate to 0.0001 per Neil Slater's suggestion, and the loss curve may have had the spikes reduced, though the scale of the graph has changed. The training loss has increased from 0.00007 to 0.00037, the validation loss from 0.0014 to 0.002, and the prediction error increased from 0.037 to 0.046.
  
  I then changed epsilon from it's value of 1e-07 to 0.1 and increased the epochs from 150 to 500. The validation loss increased to 0.0082 and the prediction error increased to 0.093, with the corresponding model loss shown below.
  
  While not an overall improvement at either step, this did remove the spikes as I requested, hence Neil's advice gives me additional considerations to explore and measure within the Adam optimizer (along with other optimizers), so I consider this to have been an important learning experience. One such exploration uncovered this more detailed explanation of optimizers than I had been exposed to before, as well as this 3D visualization of loss topologies and the effect differing optimizers and parameters have on finding the optimal minima (keep a sharp eye on the options being chosen in the upper right corner).
  
  **Tags:** python, keras, mean-squared-error
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-13. Score: 0 | Accepted answer
  
  > What might be causing these, and how to eliminate (or greatly reduce) them?
  
  It is difficult to be sure just from the graph, but I note you are using the Adam optimiser. It shares with a few other optimisers (most notably RMSProp) that it divides current gradients by a rolling mean of recent gradients to set step sizes. This can cause some minor instability when gradients get close to zero for a long while before growing again. This might happen at a saddle point where only some fraction of the network parameters are critical to results for a few iterations before hitting some other direction of slope where changing the "quiet" parameters becomes important again. This is more likely to occur as loss values become small, and close to perfect convergence.
  
  There are a couple of hyperparameters in Adam that may reduce the effect:
  
  * Reduce learning rate. The default learning rate in Adam is often set to 0.001, but you will find a lot of researchers will reduce that to magnitudes around 0.0001 or 0.00001
  * Increase epsilon. This is effectively the minimum rolling average gradient. The default is `1e-7` but it sometimes needs to be increased significantly, it depends on the loss surface. The [official documentation suggests](https://keras.io/api/optimizers/adam/) that it may even be useful to increase it up to `1.0`
  
  
  ---
  
  