Body: <p>I'm working with very weird data that is apparently very hard to fit.
  And I've noticed a very strange phenomenon where it can go from roughly 0.0176 validation MSE to 1534863.6250 validation MSE in only 1 epoch! It usually then will return to a very low number after a few epochs. Also, no such fluctuation is seen in the training data.</p>
  <p>This behavior of instability is consistent across shuffling, repartitioning &amp; retraining. Even though I have 16,000 samples and a highly regularized network (dropout + residual layers + batch normalization + gradient clipping).</p>
  <p>I mean I realize I could have more data, but, still, this behavior is really surprising. What could be causing it?</p>
  <p>P.S. Model is feedforward with 10 layers of size [32,64,128,256,512,256,128,64,32,1], using Adam optimizer. Also, <a href="https://ai.stackexchange.com/q/5471/2444">this question</a> may be related (my experience is also periodic validation loss), but I don't think they experienced the same massive instability I am seeing.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-02T14:43:45.103Z
Comments.Text: Given that you're using the MSE, I assume you're solving a regression problem, right? It may be a good idea to also share with us how much data you are using to compute the validation loss. What's the range of your inputs and targets?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-10-05T23:42:54.400Z
Comments.Text: The targets follow an exponential distribution but apparently can also be negative.
  So they can range from like -1e8 to 1e10.
Comments.UserDisplayName: profPlum
Tags: <deep-learning><overfitting><validation-loss>
Title: What can cause massive instability in validation loss?
UserDisplayName: profPlum
ThreadText: # What can cause massive instability in validation loss?
  
  (Asked by: username_0 on 2021-10-01)
  
  I'm working with very weird data that is apparently very hard to fit.
  And I've noticed a very strange phenomenon where it can go from roughly 0.0176 validation MSE to 1534863.6250 validation MSE in only 1 epoch! It usually then will return to a very low number after a few epochs. Also, no such fluctuation is seen in the training data.
  
  This behavior of instability is consistent across shuffling, repartitioning & retraining. Even though I have 16,000 samples and a highly regularized network (dropout + residual layers + batch normalization + gradient clipping).
  
  I mean I realize I could have more data, but, still, this behavior is really surprising. What could be causing it?
  
  P.S. Model is feedforward with 10 layers of size [32,64,128,256,512,256,128,64,32,1], using Adam optimizer. Also, [this question](https://ai.stackexchange.com/q/5471/2444) may be related (my experience is also periodic validation loss), but I don't think they experienced the same massive instability I am seeing.
  
  **Tags:** deep-learning, overfitting, validation-loss
  > **Comments:**
  > **username_1**: Given that you're using the MSE, I assume you're solving a regression problem, right? It may be a good idea to also share with us how much data you are using to compute the validation loss. What's the range of your inputs and targets?  
  > **username_0**: The targets follow an exponential distribution but apparently can also be negative. So they can range from like -1e8 to 1e10.  
  
  ---
  
  ## 0 answers
  
  