Body: <p>I learn a DNN for image recognition. During each epoch, I calculate mean loss in the training set. After each epoch, I calculate loss and number of errors over both training and test set. The problem is, training and test error go to (almost) zero, then increase, go to zero again, increase, and so on.  The process seems stochastic.</p>
  
  <pre><code>epoch: 1 mean_loss=0.109 train: errs=7 loss=0.00622 test: errs=3 loss=0.00608
  epoch: 2 mean_loss=0.00524 train: errs=5 loss=0.00309 test: errs=3 loss=0.00369
  epoch: 3 mean_loss=0.00408 train: errs=13 loss=0.00614 test: errs=7 loss=0.00951
  epoch: 4 mean_loss=0.00198 train: errs=113 loss=0.102 test: errs=51 loss=0.265
  epoch: 5 mean_loss=0.00424 train: errs=3 loss=0.00201 test: errs=2 loss=0.00148
  epoch: 6 mean_loss=0.0027 train: errs=1 loss=0.000466 test: errs=2 loss=0.00193
  epoch: 7 mean_loss=0.00797 train: errs=5 loss=0.00381 test: errs=0 loss=0.000493
  epoch: 8 mean_loss=0.00368 train: errs=1 loss=0.000345 test: errs=2 loss=0.00148
  epoch: 9 mean_loss=0.000358 train: errs=0 loss=6.76e-05 test: errs=0 loss=0.000446
  epoch: 10 mean_loss=0.00101 train: errs=164 loss=0.0863 test: errs=67 loss=0.19
  epoch: 11 mean_loss=0.000665 train: errs=0 loss=2.38e-05 test: errs=0 loss=9.86e-05
  epoch: 12 mean_loss=0.00714 train: errs=5 loss=0.00909 test: errs=0 loss=0.00816
  epoch: 13 mean_loss=0.00266 train: errs=73 loss=0.0333 test: errs=10 loss=0.0192
  epoch: 14 mean_loss=0.00213 train: errs=0 loss=7.74e-05 test: errs=0 loss=0.000197
  epoch: 15 mean_loss=6.12e-05 train: errs=0 loss=7.66e-05 test: errs=0 loss=3.44e-05
  epoch: 16 mean_loss=0.00162 train: errs=5 loss=0.00265 test: errs=0 loss=0.0012
  epoch: 17 mean_loss=0.000159 train: errs=0 loss=3.11e-05 test: errs=0 loss=4.26e-05
  epoch: 18 mean_loss=4.68e-05 train: errs=0 loss=3.28e-05 test: errs=0 loss=6.05e-05
  epoch: 19 mean_loss=2.47e-05 train: errs=0 loss=2.8e-05 test: errs=0 loss=5.01e-05
  epoch: 20 mean_loss=2.2e-05 train: errs=0 loss=2.31e-05 test: errs=0 loss=3.95e-05
  epoch: 21 mean_loss=2.37e-05 train: errs=0 loss=1.76e-05 test: errs=0 loss=2.52e-05
  epoch: 22 mean_loss=1.4e-05 train: errs=0 loss=1.16e-05 test: errs=0 loss=1.52e-05
  epoch: 23 mean_loss=2.13e-05 train: errs=0 loss=1.65e-05 test: errs=0 loss=2.13e-05
  epoch: 24 mean_loss=1.53e-05 train: errs=0 loss=1.91e-05 test: errs=0 loss=2.46e-05
  epoch: 25 mean_loss=0.00419 train: errs=0 loss=5.27e-05 test: errs=0 loss=4.65e-05
  epoch: 26 mean_loss=0.000372 train: errs=6 loss=0.00297 test: errs=3 loss=0.00731
  epoch: 27 mean_loss=0.0016 train: errs=0 loss=4.23e-05 test: errs=0 loss=3.69e-05
  epoch: 28 mean_loss=3.34e-05 train: errs=0 loss=2.44e-05 test: errs=0 loss=2.76e-05
  epoch: 29 mean_loss=7.03e-05 train: errs=0 loss=2.16e-05 test: errs=0 loss=1.69e-05
  epoch: 30 mean_loss=2.41e-05 train: errs=0 loss=1.84e-05 test: errs=0 loss=1.77e-05
  epoch: 31 mean_loss=1.26e-05 train: errs=0 loss=2.11e-05 test: errs=0 loss=1.78e-05
  epoch: 32 mean_loss=1.39e-05 train: errs=0 loss=2.75e-05 test: errs=0 loss=2.42e-05
  epoch: 33 mean_loss=7.68e-05 train: errs=0 loss=0.00014 test: errs=0 loss=4.66e-05
  epoch: 34 mean_loss=2.53e-05 train: errs=0 loss=1.48e-05 test: errs=0 loss=1.56e-05
  epoch: 35 mean_loss=0.000352 train: errs=1786 loss=2.17 test: errs=493 loss=2.56
  epoch: 36 mean_loss=0.0088 train: errs=0 loss=0.000347 test: errs=0 loss=0.000449
  epoch: 37 mean_loss=0.000395 train: errs=0 loss=6.18e-05 test: errs=0 loss=0.000125
  epoch: 38 mean_loss=5e-05 train: errs=0 loss=6.73e-05 test: errs=0 loss=9.89e-05
  epoch: 39 mean_loss=0.00401 train: errs=26 loss=0.00836 test: errs=27 loss=0.0269
  epoch: 40 mean_loss=0.00051 train: errs=0 loss=7.66e-05 test: errs=0 loss=7.07e-05
  epoch: 41 mean_loss=5.49e-05 train: errs=0 loss=2.47e-05 test: errs=0 loss=2.58e-05
  epoch: 42 mean_loss=3.38e-05 train: errs=0 loss=1.67e-05 test: errs=0 loss=2.1e-05
  epoch: 43 mean_loss=2.45e-05 train: errs=0 loss=1.28e-05 test: errs=0 loss=2.95e-05
  epoch: 44 mean_loss=0.00137 train: errs=44 loss=0.0141 test: errs=16 loss=0.0207
  epoch: 45 mean_loss=0.000785 train: errs=1 loss=0.000493 test: errs=0 loss=4.46e-05
  epoch: 46 mean_loss=5.46e-05 train: errs=1 loss=0.000487 test: errs=0 loss=1.34e-05
  epoch: 47 mean_loss=1.99e-05 train: errs=1 loss=0.00033 test: errs=0 loss=1.57e-05
  epoch: 48 mean_loss=1.78e-05 train: errs=1 loss=0.000307 test: errs=0 loss=1.58e-05
  epoch: 49 mean_loss=0.000903 train: errs=1 loss=0.00103 test: errs=0 loss=0.000393
  epoch: 50 mean_loss=4.74e-05 train: errs=0 loss=4.63e-05 test: errs=0 loss=3.53e-05
  Finished Training, time: 234.69774420000002 sec
  </code></pre>
  
  <p>The images are 96*96 gray.  There are about 7000 training and 1750 test images. The order of presentation is random, and different at each epoch. Each image is either contains the object or not.  The architecture is (Conv2d->ReLU->BatchNorm2d->MaxPool)*4->AvgPool(6,6)->Flatten->Conv->Conv->Conv.  All MaxPool's are 2*2. First two Conv2d layers are 5*5, padding=2, others 3*3, padding=1.  The optimiser is like this:</p>
  
  <pre><code>Optimizer= Adam (
  Parameter Group 0
      amsgrad: False
      betas: (0.9, 0.999)
      eps: 1e-08
      lr: 0.001
      weight_decay: 1e-05
  )
  </code></pre>
  
  <p>Currently I just choose the epoch when the training set error was minimal. </p>
  
  <pre><code>if epoch == 0 or train_loss &lt; train_loss_best:
      net_best = copy.deepcopy(net)
      train_loss_best = train_loss
  </code></pre>
  
  <p>It works, but I don't like it. Is there a way to make the learning more stable and steady?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T08:42:59.677Z
Comments.Text: Hi. Please, ask this question on [Data Science SE](https://datascience.stackexchange.com/questions/ask), given that involves the debugging of source code, which is usually off-topic here.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-07-16T14:15:05.173Z
Comments.Text: @nbro no, it is not about source code.  It is about the process of learning a deep NN, architecture, and learning parameters.  I am pretty sure my source code have no significant bugs, and there are only 3 lines of source code in my whole question, and I only give them to explain what exactly I am doing in some specific place.
Comments.UserDisplayName: user31264
Tags: <deep-learning><deep-neural-networks><learning-curve>
Title: Spikes in of Train and Test error
UserDisplayName: user31264
ThreadText: # Spikes in of Train and Test error
  
  (Asked by: username_0 on 2019-07-16)
  
  I learn a DNN for image recognition. During each epoch, I calculate mean loss in the training set. After each epoch, I calculate loss and number of errors over both training and test set. The problem is, training and test error go to (almost) zero, then increase, go to zero again, increase, and so on.  The process seems stochastic.
  epoch: 1 mean_loss=0.109 train: errs=7 loss=0.00622 test: errs=3 loss=0.00608
  epoch: 2 mean_loss=0.00524 train: errs=5 loss=0.00309 test: errs=3 loss=0.00369
  epoch: 3 mean_loss=0.00408 train: errs=13 loss=0.00614 test: errs=7 loss=0.00951
  epoch: 4 mean_loss=0.00198 train: errs=113 loss=0.102 test: errs=51 loss=0.265
  epoch: 5 mean_loss=0.00424 train: errs=3 loss=0.00201 test: errs=2 loss=0.00148
  epoch: 6 mean_loss=0.0027 train: errs=1 loss=0.000466 test: errs=2 loss=0.00193
  epoch: 7 mean_loss=0.00797 train: errs=5 loss=0.00381 test: errs=0 loss=0.000493
  epoch: 8 mean_loss=0.00368 train: errs=1 loss=0.000345 test: errs=2 loss=0.00148
  epoch: 9 mean_loss=0.000358 train: errs=0 loss=6.76e-05 test: errs=0 loss=0.000446
  epoch: 10 mean_loss=0.00101 train: errs=164 loss=0.0863 test: errs=67 loss=0.19
  epoch: 11 mean_loss=0.000665 train: errs=0 loss=2.38e-05 test: errs=0 loss=9.86e-05
  epoch: 12 mean_loss=0.00714 train: errs=5 loss=0.00909 test: errs=0 loss=0.00816
  epoch: 13 mean_loss=0.00266 train: errs=73 loss=0.0333 test: errs=10 loss=0.0192
  epoch: 14 mean_loss=0.00213 train: errs=0 loss=7.74e-05 test: errs=0 loss=0.000197
  epoch: 15 mean_loss=6.12e-05 train: errs=0 loss=7.66e-05 test: errs=0 loss=3.44e-05
  epoch: 16 mean_loss=0.00162 train: errs=5 loss=0.00265 test: errs=0 loss=0.0012
  epoch: 17 mean_loss=0.000159 train: errs=0 loss=3.11e-05 test: errs=0 loss=4.26e-05
  epoch: 18 mean_loss=4.68e-05 train: errs=0 loss=3.28e-05 test: errs=0 loss=6.05e-05
  epoch: 19 mean_loss=2.47e-05 train: errs=0 loss=2.8e-05 test: errs=0 loss=5.01e-05
  epoch: 20 mean_loss=2.2e-05 train: errs=0 loss=2.31e-05 test: errs=0 loss=3.95e-05
  epoch: 21 mean_loss=2.37e-05 train: errs=0 loss=1.76e-05 test: errs=0 loss=2.52e-05
  epoch: 22 mean_loss=1.4e-05 train: errs=0 loss=1.16e-05 test: errs=0 loss=1.52e-05
  epoch: 23 mean_loss=2.13e-05 train: errs=0 loss=1.65e-05 test: errs=0 loss=2.13e-05
  epoch: 24 mean_loss=1.53e-05 train: errs=0 loss=1.91e-05 test: errs=0 loss=2.46e-05
  epoch: 25 mean_loss=0.00419 train: errs=0 loss=5.27e-05 test: errs=0 loss=4.65e-05
  epoch: 26 mean_loss=0.000372 train: errs=6 loss=0.00297 test: errs=3 loss=0.00731
  epoch: 27 mean_loss=0.0016 train: errs=0 loss=4.23e-05 test: errs=0 loss=3.69e-05
  epoch: 28 mean_loss=3.34e-05 train: errs=0 loss=2.44e-05 test: errs=0 loss=2.76e-05
  epoch: 29 mean_loss=7.03e-05 train: errs=0 loss=2.16e-05 test: errs=0 loss=1.69e-05
  epoch: 30 mean_loss=2.41e-05 train: errs=0 loss=1.84e-05 test: errs=0 loss=1.77e-05
  epoch: 31 mean_loss=1.26e-05 train: errs=0 loss=2.11e-05 test: errs=0 loss=1.78e-05
  epoch: 32 mean_loss=1.39e-05 train: errs=0 loss=2.75e-05 test: errs=0 loss=2.42e-05
  epoch: 33 mean_loss=7.68e-05 train: errs=0 loss=0.00014 test: errs=0 loss=4.66e-05
  epoch: 34 mean_loss=2.53e-05 train: errs=0 loss=1.48e-05 test: errs=0 loss=1.56e-05
  epoch: 35 mean_loss=0.000352 train: errs=1786 loss=2.17 test: errs=493 loss=2.56
  epoch: 36 mean_loss=0.0088 train: errs=0 loss=0.000347 test: errs=0 loss=0.000449
  epoch: 37 mean_loss=0.000395 train: errs=0 loss=6.18e-05 test: errs=0 loss=0.000125
  epoch: 38 mean_loss=5e-05 train: errs=0 loss=6.73e-05 test: errs=0 loss=9.89e-05
  epoch: 39 mean_loss=0.00401 train: errs=26 loss=0.00836 test: errs=27 loss=0.0269
  epoch: 40 mean_loss=0.00051 train: errs=0 loss=7.66e-05 test: errs=0 loss=7.07e-05
  epoch: 41 mean_loss=5.49e-05 train: errs=0 loss=2.47e-05 test: errs=0 loss=2.58e-05
  epoch: 42 mean_loss=3.38e-05 train: errs=0 loss=1.67e-05 test: errs=0 loss=2.1e-05
  epoch: 43 mean_loss=2.45e-05 train: errs=0 loss=1.28e-05 test: errs=0 loss=2.95e-05
  epoch: 44 mean_loss=0.00137 train: errs=44 loss=0.0141 test: errs=16 loss=0.0207
  epoch: 45 mean_loss=0.000785 train: errs=1 loss=0.000493 test: errs=0 loss=4.46e-05
  epoch: 46 mean_loss=5.46e-05 train: errs=1 loss=0.000487 test: errs=0 loss=1.34e-05
  epoch: 47 mean_loss=1.99e-05 train: errs=1 loss=0.00033 test: errs=0 loss=1.57e-05
  epoch: 48 mean_loss=1.78e-05 train: errs=1 loss=0.000307 test: errs=0 loss=1.58e-05
  epoch: 49 mean_loss=0.000903 train: errs=1 loss=0.00103 test: errs=0 loss=0.000393
  epoch: 50 mean_loss=4.74e-05 train: errs=0 loss=4.63e-05 test: errs=0 loss=3.53e-05
  Finished Training, time: 234.69774420000002 sec
  
  The images are 96*96 gray.  There are about 7000 training and 1750 test images. The order of presentation is random, and different at each epoch. Each image is either contains the object or not.  The architecture is (Conv2d->ReLU->BatchNorm2d->MaxPool)*4->AvgPool(6,6)->Flatten->Conv->Conv->Conv.  All MaxPool's are 2*2. First two Conv2d layers are 5*5, padding=2, others 3*3, padding=1.  The optimiser is like this:
  Optimizer= Adam (
  Parameter Group 0
      amsgrad: False
      betas: (0.9, 0.999)
      eps: 1e-08
      lr: 0.001
      weight_decay: 1e-05
  )
  
  Currently I just choose the epoch when the training set error was minimal. 
  if epoch == 0 or train_loss < train_loss_best:
      net_best = copy.deepcopy(net)
      train_loss_best = train_loss
  
  It works, but I don't like it. Is there a way to make the learning more stable and steady?
  
  **Tags:** deep-learning, deep-neural-networks, learning-curve
  > **Comments:**
  > **username_1**: Hi. Please, ask this question on [Data Science SE](https://datascience.stackexchange.com/questions/ask), given that involves the debugging of source code, which is usually off-topic here.  
  > **username_0**: @nbro no, it is not about source code.  It is about the process of learning a deep NN, architecture, and learning parameters.  I am pretty sure my source code have no significant bugs, and there are only 3 lines of source code in my whole question, and I only give them to explain what exactly I am doing in some specific place.  
  
  ---
  
  ## 0 answers
  
  