Answers.Body: <p>The spikes could be caused by many reasons: insufficient model capacity, incorrect label, buggy input parsing, ... Finding out the culprit requires some detective work. For instance, you could apply the learned model to the whole train set and manually examine the datapoints which result in the highest loss. Alternatively, you could compare the learning outcomes of different models (both weaker and stronger). </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-02-21T14:20:03.220Z
Answers.LastActivityDate: 2019-02-21T14:20:03.220Z
Answers.UserDisplayName: ssegvic
Answers.Body: <p>Andrew Ng explains with great details in <a href="https://www.coursera.org/lecture/deep-neural-network/understanding-mini-batch-gradient-descent-lBXu8" rel="nofollow noreferrer">Deep Learning Course</a> appears in the image below.
  He also focuses on some corners can cause this problem :</p>
  
  <ol>
  <li>some mislabeled examples in the dataset.</li>
  <li>the size of your mini batch or change it with GD.</li>
  </ol>
  
  <p><a href="https://i.sstatic.net/EGqOx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/EGqOx.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T19:00:39.097Z
Answers.LastActivityDate: 2019-03-23T19:00:39.097Z
Answers.UserDisplayName: Mahmoud
Body: <p>I trained a neural network on the <a href="https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-NB15-Datasets/" rel="nofollow noreferrer">UNSW-NB15 dataset</a>, but, during training, I am getting spikes in the loss function. The algorithms see part of this UNSW dataset a single time. The loss function is plotted after every batch.</p>
  
  <p><a href="https://i.sstatic.net/yoGv8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/yoGv8.png" alt="enter image description here"></a></p>
  
  <p>For other datasets, I don't experience this problem.  I've tried different optimizers and loss functions, but this problem remains with this dataset.</p>
  
  <p>I'm using the <code>fit_generator()</code> function from Keras. Is there anyone experience this problem using Keras with this function? </p>
  
Tags: <neural-networks><machine-learning><keras><objective-functions>
Title: Why am I getting spikes in the values of the loss function during training?
UserDisplayName: A. De Rybel
ThreadText: # Why am I getting spikes in the values of the loss function during training?
  
  (Asked by: username_0 on 2019-02-20)
  
  I trained a neural network on the [UNSW-NB15 dataset](https://www.unsw.adfa.edu.au/unsw-canberra-cyber/cybersecurity/ADFA-NB15-Datasets/), but, during training, I am getting spikes in the loss function. The algorithms see part of this UNSW dataset a single time. The loss function is plotted after every batch.
  
  [![enter image description here](https://i.sstatic.net/yoGv8.png)](https://i.sstatic.net/yoGv8.png)
  
  For other datasets, I don't experience this problem. I've tried different optimizers and loss functions, but this problem remains with this dataset.
  
  I'm using the `fit_generator()` function from Keras. Is there anyone experience this problem using Keras with this function?
  
  **Tags:** neural-networks, machine-learning, keras, objective-functions
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-02-21. Score: 1
  
  The spikes could be caused by many reasons: insufficient model capacity, incorrect label, buggy input parsing, ... Finding out the culprit requires some detective work. For instance, you could apply the learned model to the whole train set and manually examine the datapoints which result in the highest loss. Alternatively, you could compare the learning outcomes of different models (both weaker and stronger).
  
  
  ---
  
  ### Answer by username_2 on 2019-03-23. Score: 1
  
  Andrew Ng explains with great details in [Deep Learning Course](https://www.coursera.org/lecture/deep-neural-network/understanding-mini-batch-gradient-descent-lBXu8) appears in the image below.
  He also focuses on some corners can cause this problem :
  
  1. some mislabeled examples in the dataset.
  2. the size of your mini batch or change it with GD.
  
  [![enter image description here](https://i.sstatic.net/EGqOx.png)](https://i.sstatic.net/EGqOx.png)
  
  
  ---
  
  