Answers.Body: <p>This is a classic &quot;exploding gradient&quot; problem. You can try adding gradient clipping or increasing the amount of regularization.</p>
  <p>Decreasing the learning rate, or increasing the batch size may also help.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T07:28:02.033Z
Answers.LastActivityDate: 2021-09-04T07:28:02.033Z
Answers.UserDisplayName: Taw
Body: <p>I'm training a Tensorflow object detection model with approx. 7500 images of two classes, which contains approx. 10,000 classes per class. I'm using Tensorflow 2.6.0, in case that is relavent. I am using Single Shot Detector (with a ResNet 50 backbone). The image dimensions are 1024 x 1024, and the batch size is set to 2. Training is being done on Ubuntu 20.04 with a GeForce RTX 2080 Super (GPU).</p>
  <p>After beginning training, the process is starting out at loss numbers to be expected:</p>
  <pre><code>INFO:tensorflow:{'Loss/classification_loss': 2.1305692,
   'Loss/localization_loss': 0.6402807,
   'Loss/regularization_loss': 1.407957,
   'Loss/total_loss': 4.178807,
   'learning_rate': 0.014666351}
  I0903 16:56:21.947736 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 2.1305692,
   'Loss/localization_loss': 0.6402807,
   'Loss/regularization_loss': 1.407957,
   'Loss/total_loss': 4.178807,
   'learning_rate': 0.014666351}
  INFO:tensorflow:Step 200 per-step time 0.447s
  I0903 16:57:06.592366 140581900665792 model_lib_v2.py:698] Step 200 per-step time 0.447s
  INFO:tensorflow:{'Loss/classification_loss': 1.2596315,
   'Loss/localization_loss': 0.6752764,
   'Loss/regularization_loss': 3.0123177,
   'Loss/total_loss': 4.9472256,
   'learning_rate': 0.0159997}
  I0903 16:57:06.592768 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.2596315,
   'Loss/localization_loss': 0.6752764,
   'Loss/regularization_loss': 3.0123177,
   'Loss/total_loss': 4.9472256,
   'learning_rate': 0.0159997}
  INFO:tensorflow:Step 300 per-step time 0.452s
  I0903 16:57:51.830375 140581900665792 model_lib_v2.py:698] Step 300 per-step time 0.452s
  INFO:tensorflow:{'Loss/classification_loss': 1.0455683,
   'Loss/localization_loss': 0.5895866,
   'Loss/regularization_loss': 3.0799737,
   'Loss/total_loss': 4.715129,
   'learning_rate': 0.01733305}
  I0903 16:57:51.830749 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.0455683,
   'Loss/localization_loss': 0.5895866,
   'Loss/regularization_loss': 3.0799737,
   'Loss/total_loss': 4.715129,
   'learning_rate': 0.01733305}
  </code></pre>
  <p>Up until about step 16,800, the loss is decreasing to these numbers:</p>
  <pre><code>INFO:tensorflow:{'Loss/classification_loss': 0.5526215,
   'Loss/localization_loss': 0.28333753,
   'Loss/regularization_loss': 0.24686696,
   'Loss/total_loss': 1.0828259,
   'learning_rate': 0.037849143}
  I0903 18:59:14.666097 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.5526215,
   'Loss/localization_loss': 0.28333753,
   'Loss/regularization_loss': 0.24686696,
   'Loss/total_loss': 1.0828259,
   'learning_rate': 0.037849143}
  INFO:tensorflow:Step 16700 per-step time 0.446s
  I0903 18:59:59.247199 140581900665792 model_lib_v2.py:698] Step 16700 per-step time 0.446s
  INFO:tensorflow:{'Loss/classification_loss': 0.4649979,
   'Loss/localization_loss': 0.28323257,
   'Loss/regularization_loss': 0.2433301,
   'Loss/total_loss': 0.9915606,
   'learning_rate': 0.037820127}
  I0903 18:59:59.247609 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.4649979,
   'Loss/localization_loss': 0.28323257,
   'Loss/regularization_loss': 0.2433301,
   'Loss/total_loss': 0.9915606,
   'learning_rate': 0.037820127}
  INFO:tensorflow:Step 16800 per-step time 0.446s
  I0903 19:00:43.835976 140581900665792 model_lib_v2.py:698] Step 16800 per-step time 0.446s
  INFO:tensorflow:{'Loss/classification_loss': 0.43402833,
   'Loss/localization_loss': 0.1641234,
   'Loss/regularization_loss': 0.24129395,
   'Loss/total_loss': 0.8394457,
   'learning_rate': 0.03779093}
  I0903 19:00:43.836373 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.43402833,
   'Loss/localization_loss': 0.1641234,
   'Loss/regularization_loss': 0.24129395,
   'Loss/total_loss': 0.8394457,
   'learning_rate': 0.03779093}
  </code></pre>
  <p>However, starting at about 16,900, the model <code>total_loss</code> rapidly increases, up to numbers even higher than are shown below:</p>
  <pre><code>INFO:tensorflow:Step 16900 per-step time 0.446s
  I0903 19:01:28.390861 140581900665792 model_lib_v2.py:698] Step 16900 per-step time 0.446s
  INFO:tensorflow:{'Loss/classification_loss': 0.5590624,
   'Loss/localization_loss': 0.5160909,
   'Loss/regularization_loss': 338.40286,
   'Loss/total_loss': 339.478,
   'learning_rate': 0.03776155}
  I0903 19:01:28.391232 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.5590624,
   'Loss/localization_loss': 0.5160909,
   'Loss/regularization_loss': 338.40286,
   'Loss/total_loss': 339.478,
   'learning_rate': 0.03776155}
  INFO:tensorflow:Step 17000 per-step time 0.445s
  I0903 19:02:12.936022 140581900665792 model_lib_v2.py:698] Step 17000 per-step time 0.445s
  INFO:tensorflow:{'Loss/classification_loss': 0.7908556,
   'Loss/localization_loss': 0.7274248,
   'Loss/regularization_loss': 858.3554,
   'Loss/total_loss': 859.87366,
   'learning_rate': 0.037731986}
  I0903 19:02:12.936432 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.7908556,
   'Loss/localization_loss': 0.7274248,
   'Loss/regularization_loss': 858.3554,
   'Loss/total_loss': 859.87366,
   'learning_rate': 0.037731986}
  INFO:tensorflow:Step 17100 per-step time 0.452s
  I0903 19:02:58.127156 140581900665792 model_lib_v2.py:698] Step 17100 per-step time 0.452s
  INFO:tensorflow:{'Loss/classification_loss': 0.7510178,
   'Loss/localization_loss': 0.49337074,
   'Loss/regularization_loss': 2617.2888,
   'Loss/total_loss': 2618.5332,
   'learning_rate': 0.03770224}
  I0903 19:02:58.127575 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.7510178,
   'Loss/localization_loss': 0.49337074,
   'Loss/regularization_loss': 2617.2888,
   'Loss/total_loss': 2618.5332,
   'learning_rate': 0.03770224}
  INFO:tensorflow:Step 17200 per-step time 0.445s
  I0903 19:03:42.625258 140581900665792 model_lib_v2.py:698] Step 17200 per-step time 0.445s
  INFO:tensorflow:{'Loss/classification_loss': 1.1258743,
   'Loss/localization_loss': 0.45634705,
   'Loss/regularization_loss': 394886900.0,
   'Loss/total_loss': 394886900.0,
   'learning_rate': 0.037672307}
  I0903 19:03:42.625638 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.1258743,
   'Loss/localization_loss': 0.45634705,
   'Loss/regularization_loss': 394886900.0,
   'Loss/total_loss': 394886900.0,
   'learning_rate': 0.037672307}
  INFO:tensorflow:Step 17300 per-step time 0.445s
  I0903 19:04:27.112154 140581900665792 model_lib_v2.py:698] Step 17300 per-step time 0.445s
  INFO:tensorflow:{'Loss/classification_loss': 0.57859087,
   'Loss/localization_loss': 0.53405523,
   'Loss/regularization_loss': 383440770.0,
   'Loss/total_loss': 383440770.0,
   'learning_rate': 0.037642203}
  I0903 19:04:27.112533 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.57859087,
   'Loss/localization_loss': 0.53405523,
   'Loss/regularization_loss': 383440770.0,
   'Loss/total_loss': 383440770.0,
   'learning_rate': 0.037642203}
  </code></pre>
  <p>What could be the cause of this, and what would be the best way to go about fixing it?</p>
  
Tags: <deep-learning><tensorflow><training><object-detection>
Title: Tensorflow object detection model total loss starts out good, but suddenly explodes up to high loss numbers
UserDisplayName: sneeze_shiny
ThreadText: # Tensorflow object detection model total loss starts out good, but suddenly explodes up to high loss numbers
  
  (Asked by: username_0 on 2021-09-04)
  
  I'm training a Tensorflow object detection model with approx. 7500 images of two classes, which contains approx. 10,000 classes per class. I'm using Tensorflow 2.6.0, in case that is relavent. I am using Single Shot Detector (with a ResNet 50 backbone). The image dimensions are 1024 x 1024, and the batch size is set to 2. Training is being done on Ubuntu 20.04 with a GeForce RTX 2080 Super (GPU).
  After beginning training, the process is starting out at loss numbers to be expected:
  INFO:tensorflow:{'Loss/classification_loss': 2.1305692,
   'Loss/localization_loss': 0.6402807,
   'Loss/regularization_loss': 1.407957,
   'Loss/total_loss': 4.178807,
   'learning_rate': 0.014666351}
  I0903 16:56:21.947736 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 2.1305692,
   'Loss/localization_loss': 0.6402807,
   'Loss/regularization_loss': 1.407957,
   'Loss/total_loss': 4.178807,
   'learning_rate': 0.014666351}
  INFO:tensorflow:Step 200 per-step time 0.447s
  I0903 16:57:06.592366 140581900665792 model_lib_v2.py:698] Step 200 per-step time 0.447s
  INFO:tensorflow:{'Loss/classification_loss': 1.2596315,
   'Loss/localization_loss': 0.6752764,
   'Loss/regularization_loss': 3.0123177,
   'Loss/total_loss': 4.9472256,
   'learning_rate': 0.0159997}
  I0903 16:57:06.592768 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.2596315,
   'Loss/localization_loss': 0.6752764,
   'Loss/regularization_loss': 3.0123177,
   'Loss/total_loss': 4.9472256,
   'learning_rate': 0.0159997}
  INFO:tensorflow:Step 300 per-step time 0.452s
  I0903 16:57:51.830375 140581900665792 model_lib_v2.py:698] Step 300 per-step time 0.452s
  INFO:tensorflow:{'Loss/classification_loss': 1.0455683,
   'Loss/localization_loss': 0.5895866,
   'Loss/regularization_loss': 3.0799737,
   'Loss/total_loss': 4.715129,
   'learning_rate': 0.01733305}
  I0903 16:57:51.830749 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.0455683,
   'Loss/localization_loss': 0.5895866,
   'Loss/regularization_loss': 3.0799737,
   'Loss/total_loss': 4.715129,
   'learning_rate': 0.01733305}
  
  Up until about step 16,800, the loss is decreasing to these numbers:
  INFO:tensorflow:{'Loss/classification_loss': 0.5526215,
   'Loss/localization_loss': 0.28333753,
   'Loss/regularization_loss': 0.24686696,
   'Loss/total_loss': 1.0828259,
   'learning_rate': 0.037849143}
  I0903 18:59:14.666097 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.5526215,
   'Loss/localization_loss': 0.28333753,
   'Loss/regularization_loss': 0.24686696,
   'Loss/total_loss': 1.0828259,
   'learning_rate': 0.037849143}
  INFO:tensorflow:Step 16700 per-step time 0.446s
  I0903 18:59:59.247199 140581900665792 model_lib_v2.py:698] Step 16700 per-step time 0.446s
  INFO:tensorflow:{'Loss/classification_loss': 0.4649979,
   'Loss/localization_loss': 0.28323257,
   'Loss/regularization_loss': 0.2433301,
   'Loss/total_loss': 0.9915606,
   'learning_rate': 0.037820127}
  I0903 18:59:59.247609 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.4649979,
   'Loss/localization_loss': 0.28323257,
   'Loss/regularization_loss': 0.2433301,
   'Loss/total_loss': 0.9915606,
   'learning_rate': 0.037820127}
  INFO:tensorflow:Step 16800 per-step time 0.446s
  I0903 19:00:43.835976 140581900665792 model_lib_v2.py:698] Step 16800 per-step time 0.446s
  INFO:tensorflow:{'Loss/classification_loss': 0.43402833,
   'Loss/localization_loss': 0.1641234,
   'Loss/regularization_loss': 0.24129395,
   'Loss/total_loss': 0.8394457,
   'learning_rate': 0.03779093}
  I0903 19:00:43.836373 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.43402833,
   'Loss/localization_loss': 0.1641234,
   'Loss/regularization_loss': 0.24129395,
   'Loss/total_loss': 0.8394457,
   'learning_rate': 0.03779093}
  
  However, starting at about 16,900, the model total_loss rapidly increases, up to numbers even higher than are shown below:
  INFO:tensorflow:Step 16900 per-step time 0.446s
  I0903 19:01:28.390861 140581900665792 model_lib_v2.py:698] Step 16900 per-step time 0.446s
  INFO:tensorflow:{'Loss/classification_loss': 0.5590624,
   'Loss/localization_loss': 0.5160909,
   'Loss/regularization_loss': 338.40286,
   'Loss/total_loss': 339.478,
   'learning_rate': 0.03776155}
  I0903 19:01:28.391232 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.5590624,
   'Loss/localization_loss': 0.5160909,
   'Loss/regularization_loss': 338.40286,
   'Loss/total_loss': 339.478,
   'learning_rate': 0.03776155}
  INFO:tensorflow:Step 17000 per-step time 0.445s
  I0903 19:02:12.936022 140581900665792 model_lib_v2.py:698] Step 17000 per-step time 0.445s
  INFO:tensorflow:{'Loss/classification_loss': 0.7908556,
   'Loss/localization_loss': 0.7274248,
   'Loss/regularization_loss': 858.3554,
   'Loss/total_loss': 859.87366,
   'learning_rate': 0.037731986}
  I0903 19:02:12.936432 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.7908556,
   'Loss/localization_loss': 0.7274248,
   'Loss/regularization_loss': 858.3554,
   'Loss/total_loss': 859.87366,
   'learning_rate': 0.037731986}
  INFO:tensorflow:Step 17100 per-step time 0.452s
  I0903 19:02:58.127156 140581900665792 model_lib_v2.py:698] Step 17100 per-step time 0.452s
  INFO:tensorflow:{'Loss/classification_loss': 0.7510178,
   'Loss/localization_loss': 0.49337074,
   'Loss/regularization_loss': 2617.2888,
   'Loss/total_loss': 2618.5332,
   'learning_rate': 0.03770224}
  I0903 19:02:58.127575 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.7510178,
   'Loss/localization_loss': 0.49337074,
   'Loss/regularization_loss': 2617.2888,
   'Loss/total_loss': 2618.5332,
   'learning_rate': 0.03770224}
  INFO:tensorflow:Step 17200 per-step time 0.445s
  I0903 19:03:42.625258 140581900665792 model_lib_v2.py:698] Step 17200 per-step time 0.445s
  INFO:tensorflow:{'Loss/classification_loss': 1.1258743,
   'Loss/localization_loss': 0.45634705,
   'Loss/regularization_loss': 394886900.0,
   'Loss/total_loss': 394886900.0,
   'learning_rate': 0.037672307}
  I0903 19:03:42.625638 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 1.1258743,
   'Loss/localization_loss': 0.45634705,
   'Loss/regularization_loss': 394886900.0,
   'Loss/total_loss': 394886900.0,
   'learning_rate': 0.037672307}
  INFO:tensorflow:Step 17300 per-step time 0.445s
  I0903 19:04:27.112154 140581900665792 model_lib_v2.py:698] Step 17300 per-step time 0.445s
  INFO:tensorflow:{'Loss/classification_loss': 0.57859087,
   'Loss/localization_loss': 0.53405523,
   'Loss/regularization_loss': 383440770.0,
   'Loss/total_loss': 383440770.0,
   'learning_rate': 0.037642203}
  I0903 19:04:27.112533 140581900665792 model_lib_v2.py:701] {'Loss/classification_loss': 0.57859087,
   'Loss/localization_loss': 0.53405523,
   'Loss/regularization_loss': 383440770.0,
   'Loss/total_loss': 383440770.0,
   'learning_rate': 0.037642203}
  
  What could be the cause of this, and what would be the best way to go about fixing it?
  
  **Tags:** deep-learning, tensorflow, training, object-detection
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-09-04. Score: 1 | Accepted answer
  
  This is a classic "exploding gradient" problem. You can try adding gradient clipping or increasing the amount of regularization.
  
  Decreasing the learning rate, or increasing the batch size may also help.
  
  
  ---
  
  