Answers.Body: <p>very interesting questions:</p>
  <p><strong>1. what exactly is happening when training and validation accuracy change during training</strong></p>
  <ul>
  <li>The <code>accuracy</code> change after every batch computation. You have 588 batches, so loss will be computed after each one of these batches (let's say each batch have 8 images). However, the accuracy you see in the progress bar it is the accuracy of the current batch <em>averaged</em> with the accuracy of all the previous batches so far. See <code>keras.utils.generic_utils.Progbar</code>.</li>
  <li>The <code>val_acc</code> is computed only at the end of one epoch and it is computed with all your validation dataset at once (considering it as a single batch, so if you have 100 images for validation it will compute accuracy as a single batch of 100 images)</li>
  </ul>
  <p><strong>2. what do different behaviours imply</strong></p>
  <ul>
  <li><p>The <code>acc</code> and <code>val_acc</code> normally differ from each other due to different split sizes.</p>
  <ul>
  <li>Try same experiment with <code>validation_split=0.01</code> and <code>validation_split=0.4</code> and you will see how both <code>accuracy</code> and <code>val_acc</code> will change.</li>
  <li>Normally the greater the validation split, the more similar both metrics will be since the validation split will be big enough to be representative (let's say it has cats and dogs, not only cats), taking into account that you need enough data to train correctly. This explains why in some cases the <code>val_acc</code> is higher than <code>accuracy</code> and vice versa.</li>
  </ul>
  </li>
  <li><p>Overfitting only occurs when the graph fashion or tendency changes and <code>val_acc</code> starts to drop and <code>accuracy</code> keeping increasing. This means that your model can not do any better with the validation dataset (non previously seen images).</p>
  </li>
  </ul>
  <p>I work with <code>loss</code> and <code>val_loss</code> which are highly correlated with <code>accuracy</code>. Normally the loss is the inverse, so interpret the comments above in the inverse sense (sorry about the confusion but I'm taking this example from my current experiments) I hope it helps:</p>
  <p><a href="https://i.sstatic.net/zvku1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zvku1.png" alt="enter image description here" /></a></p>
  <p>There are 2 experiments, orange and grey.</p>
  <ul>
  <li><p>In both experiments, <code>val_loss</code> is always slightly higher than <code>loss</code> (because of my current validation split which it happens to be also 0.2, but normally is 0.01 and val_loss is even higher).</p>
  </li>
  <li><p>On both experiments the <code>loss</code> trend is linearly decreasing, this is because gradient descent works and the loss functions is well defined and it converges.</p>
  </li>
  <li><p>Orange experiment is overfitting from epoch 20 onwards because, the <code>val_loss</code> won't drop any more and, on the contrary, it start increasing.</p>
  </li>
  <li><p>Grey experiment is just right, both <code>loss</code> and <code>val_loss</code> are still decreasing, and although the <code>val_loss</code> might be greater than <code>loss</code> it is not overfitting because it is still decreasing. So that is why it is still training :)</p>
  </li>
  </ul>
  <p>Complex concepts here, I hope I was able to explain myself clearly! Cheers</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-04T12:21:10.897Z
Answers.Text: Thank you for the detailed explanation! Just a few questions: 1) If loss is computed after each batch and `val_acc` is computed only at the end of one epoch, does `val_acc`/`accuracy` reflect the accuracy of the entire model or only the batch? 2) Acc vs loss: when one is increasing, does the other necessarily have to decrease proportionally? Thank you
Answers.UserDisplayName: nilsinelabore
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-04T13:33:22.827Z
Answers.Text: 1) In Keras after computing the `accuracy` for the batch, it is averaged with the accuracy from all the previous batches, so yes, it takes them all into account and hence it is representative for the accuracy of your model. Same with `val_acc` (but with only one batch)
  2) In 99% of cases yes, but it depends on how you define loss and accuracy (in standard keras metrics they are inversely proportional)
Answers.UserDisplayName: JVGD
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-04T13:41:57.663Z
Answers.Text: I edited the response to add this
Answers.UserDisplayName: JVGD
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-23T10:14:51.977Z
Answers.Text: Also note that using `mode.evaluate` to the training data may give you a different loss than what is printed during `model.fit`, since `fit` calculates the loss after each epoch separately and the model's parameters are being updated in-between.
Answers.UserDisplayName: NikoNyrh
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-04T10:18:26.693Z
Answers.LastActivityDate: 2022-05-22T21:42:33.110Z
Answers.LastEditDate: 2022-05-22T21:42:33.110Z
Answers.UserDisplayName: JVGD
Body: <p>During model training, I noticed various behaviour in between training and validation accuracy. I understand that <a href="https://stackoverflow.com/a/51345413/11901732">'The training set is used to train the model, while the validation set is only used to evaluate the model's performance...'</a>, but I'd like to know if there is any relationship between training and validation accuracy and, if yes,</p>
  <ol>
  <li><p><strong>what exactly is happening when training and validation accuracy change during training and;</strong></p>
  </li>
  <li><p><strong>what do different behaviours imply</strong></p>
  </li>
  </ol>
  <p>For instance, some believe there is overfitting problem if training &gt; validation accuracy. What happens if one is greater than the other <strong>alternately</strong>, which is the case below?</p>
  <p>Here is the code</p>
  <pre><code>inputs_1 = keras.Input(shape=(10081,1))
  
  layer1 = Conv1D(64,14)(inputs_1)
  layer2 = layers.MaxPool1D(5)(layer1)
  layer3 = Conv1D(64, 14)(layer2)
  layer4 = layers.GlobalMaxPooling1D()(layer3)
  
  inputs_2 = keras.Input(shape=(104,))             
  layer5 = layers.concatenate([layer4, inputs_2])
  layer6 = Dense(128, activation='relu')(layer5)
  layer7 = Dense(2, activation='softmax')(layer6)
  
  
  model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], output = [layer7])
  model_2.summary()
  
  
  X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:10185], df[['Result_cat','Result_cat1']].values, test_size=0.2) 
  X_train = X_train.to_numpy()
  X_train = X_train.reshape([X_train.shape[0], X_train.shape[1], 1]) 
  X_train_1 = X_train[:,0:10081,:]
  X_train_2 = X_train[:,10081:10185,:].reshape(736,104)   
  
  
  X_test = X_test.to_numpy()
  X_test = X_test.reshape([X_test.shape[0], X_test.shape[1], 1]) 
  X_test_1 = X_test[:,0:10081,:]
  X_test_2 = X_test[:,10081:10185,:].reshape(185,104)    
  
  adam = keras.optimizers.Adam(lr = 0.0005)
  model_2.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['acc'])
  
  history = model_2.fit([X_train_1,X_train_2], y_train, epochs = 120, batch_size = 256, validation_split = 0.2, callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)])
  </code></pre>
  <p>model summary</p>
  <pre><code>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:15: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=[&lt;tf.Tenso..., outputs=[&lt;tf.Tenso...)`
    from ipykernel import kernelapp as app
  Model: &quot;model_3&quot;
  __________________________________________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_5 (InputLayer)            (None, 10081, 1)     0                                            
  __________________________________________________________________________________________________
  conv1d_5 (Conv1D)               (None, 10068, 64)    960         input_5[0][0]                    
  __________________________________________________________________________________________________
  max_pooling1d_3 (MaxPooling1D)  (None, 2013, 64)     0           conv1d_5[0][0]                   
  __________________________________________________________________________________________________
  conv1d_6 (Conv1D)               (None, 2000, 64)     57408       max_pooling1d_3[0][0]            
  __________________________________________________________________________________________________
  global_max_pooling1d_3 (GlobalM (None, 64)           0           conv1d_6[0][0]                   
  __________________________________________________________________________________________________
  input_6 (InputLayer)            (None, 104)          0                                            
  __________________________________________________________________________________________________
  concatenate_3 (Concatenate)     (None, 168)          0           global_max_pooling1d_3[0][0]     
                                                                   input_6[0][0]                    
  __________________________________________________________________________________________________
  dense_5 (Dense)                 (None, 128)          21632       concatenate_3[0][0]              
  __________________________________________________________________________________________________
  dense_6 (Dense)                 (None, 2)            258         dense_5[0][0]                    
  ==================================================================================================
  Total params: 80,258
  Trainable params: 80,258
  Non-trainable params: 0
  </code></pre>
  <p>and the training process</p>
  <pre><code>__________________________________________________________________________________________________
  Train on 588 samples, validate on 148 samples
  Epoch 1/120
  588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
  Epoch 2/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
  Epoch 3/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
  Epoch 4/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
  Epoch 5/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
  Epoch 6/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
  Epoch 7/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
  Epoch 8/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
  Epoch 9/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
  Epoch 10/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
  Epoch 11/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
  Epoch 12/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
  Epoch 13/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
  Epoch 14/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
  Epoch 15/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
  Epoch 16/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
  Epoch 17/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
  Epoch 18/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
  Epoch 19/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
  Epoch 20/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
  Epoch 21/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
  Epoch 22/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
  Epoch 23/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
  Epoch 24/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
  Epoch 25/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
  Epoch 26/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
  Epoch 27/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
  Epoch 28/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
  Epoch 29/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
  Epoch 30/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
  Epoch 31/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
  Epoch 32/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
  Epoch 33/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
  Epoch 34/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
  Epoch 35/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
  Epoch 36/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
  Epoch 37/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
  Epoch 38/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
  Epoch 39/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
  Epoch 40/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
  Epoch 41/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
  Epoch 42/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
  Epoch 43/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
  Epoch 44/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
  Epoch 45/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
  Epoch 46/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
  Epoch 47/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
  Epoch 48/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
  Epoch 49/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
  Epoch 50/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
  Epoch 51/120
  588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
  Epoch 52/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
  Epoch 53/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
  Epoch 54/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
  Epoch 55/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
  Epoch 56/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
  Epoch 57/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
  Epoch 58/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
  Epoch 59/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
  Epoch 60/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
  Epoch 61/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
  Epoch 62/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
  Epoch 63/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
  Epoch 64/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
  Epoch 65/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
  Epoch 66/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
  Epoch 67/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
  Epoch 68/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
  Epoch 69/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
  Epoch 70/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
  Epoch 71/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041__________________________________________________________________________________________________
  Train on 588 samples, validate on 148 samples
  Epoch 1/120
  588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
  Epoch 2/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
  Epoch 3/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
  Epoch 4/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
  Epoch 5/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
  Epoch 6/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
  Epoch 7/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
  Epoch 8/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
  Epoch 9/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
  Epoch 10/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
  Epoch 11/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
  Epoch 12/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
  Epoch 13/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
  Epoch 14/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
  Epoch 15/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
  Epoch 16/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
  Epoch 17/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
  Epoch 18/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
  Epoch 19/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
  Epoch 20/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
  Epoch 21/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
  Epoch 22/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
  Epoch 23/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
  Epoch 24/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
  Epoch 25/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
  Epoch 26/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
  Epoch 27/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
  Epoch 28/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
  Epoch 29/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
  Epoch 30/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
  Epoch 31/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
  Epoch 32/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
  Epoch 33/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
  Epoch 34/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
  Epoch 35/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
  Epoch 36/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
  Epoch 37/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
  Epoch 38/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
  Epoch 39/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
  Epoch 40/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
  Epoch 41/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
  Epoch 42/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
  Epoch 43/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
  Epoch 44/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
  Epoch 45/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
  Epoch 46/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
  Epoch 47/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
  Epoch 48/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
  Epoch 49/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
  Epoch 50/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
  Epoch 51/120
  588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
  Epoch 52/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
  Epoch 53/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
  Epoch 54/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
  Epoch 55/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
  Epoch 56/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
  Epoch 57/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
  Epoch 58/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
  Epoch 59/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
  Epoch 60/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
  Epoch 61/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
  Epoch 62/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
  Epoch 63/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
  Epoch 64/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
  Epoch 65/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
  Epoch 66/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
  Epoch 67/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
  Epoch 68/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
  Epoch 69/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
  Epoch 70/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
  Epoch 71/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041
  </code></pre>
  <p>Notice how at first <code>acc</code> is lower than <code>val_acc</code> and later is greater than <code>val_acc</code>. Can someone please shed some light what could be happening here? Thank you</p>
  
Tags: <neural-networks><machine-learning><training><accuracy><cross-validation>
Title: What is the relationship between the training accuracy and validation accuracy?
UserDisplayName: nilsinelabore
ThreadText: # What is the relationship between the training accuracy and validation accuracy?
  
  (Asked by: username_0 on 2019-12-03)
  
  During model training, I noticed various behaviour in between training and validation accuracy. I understand that ['The training set is used to train the model, while the validation set is only used to evaluate the model's performance...'](https://stackoverflow.com/a/51345413/11901732), but I'd like to know if there is any relationship between training and validation accuracy and, if yes,
  
  1. **what exactly is happening when training and validation accuracy change during training and;**
  2. **what do different behaviours imply**
  
  For instance, some believe there is overfitting problem if training > validation accuracy. What happens if one is greater than the other **alternately**, which is the case below?
  
  Here is the code
  
  ```
  inputs_1 = keras.Input(shape=(10081,1))
  
  layer1 = Conv1D(64,14)(inputs_1)
  layer2 = layers.MaxPool1D(5)(layer1)
  layer3 = Conv1D(64, 14)(layer2)
  layer4 = layers.GlobalMaxPooling1D()(layer3)
  
  inputs_2 = keras.Input(shape=(104,))             
  layer5 = layers.concatenate([layer4, inputs_2])
  layer6 = Dense(128, activation='relu')(layer5)
  layer7 = Dense(2, activation='softmax')(layer6)
  
  model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], output = [layer7])
  model_2.summary()
  
  X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:10185], df[['Result_cat','Result_cat1']].values, test_size=0.2) 
  X_train = X_train.to_numpy()
  X_train = X_train.reshape([X_train.shape[0], X_train.shape[1], 1]) 
  X_train_1 = X_train[:,0:10081,:]
  X_train_2 = X_train[:,10081:10185,:].reshape(736,104)   
  
  X_test = X_test.to_numpy()
  X_test = X_test.reshape([X_test.shape[0], X_test.shape[1], 1]) 
  X_test_1 = X_test[:,0:10081,:]
  X_test_2 = X_test[:,10081:10185,:].reshape(185,104)    
  
  adam = keras.optimizers.Adam(lr = 0.0005)
  model_2.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['acc'])
  
  history = model_2.fit([X_train_1,X_train_2], y_train, epochs = 120, batch_size = 256, validation_split = 0.2, callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)])
  
  ```
  
  model summary
  
  ```
  /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:15: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=[<tf.Tenso..., outputs=[<tf.Tenso...)`
    from ipykernel import kernelapp as app
  Model: "model_3"
  __________________________________________________________________________________________________
  Layer (type)                    Output Shape         Param #     Connected to                     
  ==================================================================================================
  input_5 (InputLayer)            (None, 10081, 1)     0                                            
  __________________________________________________________________________________________________
  conv1d_5 (Conv1D)               (None, 10068, 64)    960         input_5[0][0]                    
  __________________________________________________________________________________________________
  max_pooling1d_3 (MaxPooling1D)  (None, 2013, 64)     0           conv1d_5[0][0]                   
  __________________________________________________________________________________________________
  conv1d_6 (Conv1D)               (None, 2000, 64)     57408       max_pooling1d_3[0][0]            
  __________________________________________________________________________________________________
  global_max_pooling1d_3 (GlobalM (None, 64)           0           conv1d_6[0][0]                   
  __________________________________________________________________________________________________
  input_6 (InputLayer)            (None, 104)          0                                            
  __________________________________________________________________________________________________
  concatenate_3 (Concatenate)     (None, 168)          0           global_max_pooling1d_3[0][0]     
                                                                   input_6[0][0]                    
  __________________________________________________________________________________________________
  dense_5 (Dense)                 (None, 128)          21632       concatenate_3[0][0]              
  __________________________________________________________________________________________________
  dense_6 (Dense)                 (None, 2)            258         dense_5[0][0]                    
  ==================================================================================================
  Total params: 80,258
  Trainable params: 80,258
  Non-trainable params: 0
  
  ```
  
  and the training process
  
  ```
  __________________________________________________________________________________________________
  Train on 588 samples, validate on 148 samples
  Epoch 1/120
  588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
  Epoch 2/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
  Epoch 3/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
  Epoch 4/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
  Epoch 5/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
  Epoch 6/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
  Epoch 7/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
  Epoch 8/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
  Epoch 9/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
  Epoch 10/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
  Epoch 11/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
  Epoch 12/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
  Epoch 13/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
  Epoch 14/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
  Epoch 15/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
  Epoch 16/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
  Epoch 17/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
  Epoch 18/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
  Epoch 19/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
  Epoch 20/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
  Epoch 21/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
  Epoch 22/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
  Epoch 23/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
  Epoch 24/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
  Epoch 25/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
  Epoch 26/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
  Epoch 27/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
  Epoch 28/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
  Epoch 29/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
  Epoch 30/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
  Epoch 31/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
  Epoch 32/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
  Epoch 33/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
  Epoch 34/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
  Epoch 35/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
  Epoch 36/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
  Epoch 37/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
  Epoch 38/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
  Epoch 39/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
  Epoch 40/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
  Epoch 41/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
  Epoch 42/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
  Epoch 43/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
  Epoch 44/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
  Epoch 45/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
  Epoch 46/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
  Epoch 47/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
  Epoch 48/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
  Epoch 49/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
  Epoch 50/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
  Epoch 51/120
  588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
  Epoch 52/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
  Epoch 53/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
  Epoch 54/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
  Epoch 55/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
  Epoch 56/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
  Epoch 57/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
  Epoch 58/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
  Epoch 59/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
  Epoch 60/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
  Epoch 61/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
  Epoch 62/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
  Epoch 63/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
  Epoch 64/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
  Epoch 65/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
  Epoch 66/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
  Epoch 67/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
  Epoch 68/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
  Epoch 69/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
  Epoch 70/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
  Epoch 71/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041__________________________________________________________________________________________________
  Train on 588 samples, validate on 148 samples
  Epoch 1/120
  588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
  Epoch 2/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
  Epoch 3/120
  588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
  Epoch 4/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
  Epoch 5/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
  Epoch 6/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
  Epoch 7/120
  588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
  Epoch 8/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
  Epoch 9/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
  Epoch 10/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
  Epoch 11/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
  Epoch 12/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
  Epoch 13/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
  Epoch 14/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
  Epoch 15/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
  Epoch 16/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
  Epoch 17/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
  Epoch 18/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
  Epoch 19/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
  Epoch 20/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
  Epoch 21/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
  Epoch 22/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
  Epoch 23/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
  Epoch 24/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
  Epoch 25/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
  Epoch 26/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
  Epoch 27/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
  Epoch 28/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
  Epoch 29/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
  Epoch 30/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
  Epoch 31/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
  Epoch 32/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
  Epoch 33/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
  Epoch 34/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
  Epoch 35/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
  Epoch 36/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
  Epoch 37/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
  Epoch 38/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
  Epoch 39/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
  Epoch 40/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
  Epoch 41/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
  Epoch 42/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
  Epoch 43/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
  Epoch 44/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
  Epoch 45/120
  588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
  Epoch 46/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
  Epoch 47/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
  Epoch 48/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
  Epoch 49/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
  Epoch 50/120
  588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
  Epoch 51/120
  588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
  Epoch 52/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
  Epoch 53/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
  Epoch 54/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
  Epoch 55/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
  Epoch 56/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
  Epoch 57/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
  Epoch 58/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
  Epoch 59/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
  Epoch 60/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
  Epoch 61/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
  Epoch 62/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
  Epoch 63/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
  Epoch 64/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
  Epoch 65/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
  Epoch 66/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
  Epoch 67/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
  Epoch 68/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
  Epoch 69/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
  Epoch 70/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
  Epoch 71/120
  588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041
  
  ```
  
  Notice how at first `acc` is lower than `val_acc` and later is greater than `val_acc`. Can someone please shed some light what could be happening here? Thank you
  
  **Tags:** neural-networks, machine-learning, training, accuracy, cross-validation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-12-04. Score: 1 | Accepted answer
  
  very interesting questions:
  
  **1. what exactly is happening when training and validation accuracy change during training**
  
  * The `accuracy` change after every batch computation. You have 588 batches, so loss will be computed after each one of these batches (let's say each batch have 8 images). However, the accuracy you see in the progress bar it is the accuracy of the current batch *averaged* with the accuracy of all the previous batches so far. See `keras.utils.generic_utils.Progbar`.
  * The `val_acc` is computed only at the end of one epoch and it is computed with all your validation dataset at once (considering it as a single batch, so if you have 100 images for validation it will compute accuracy as a single batch of 100 images)
  
  **2. what do different behaviours imply**
  
  * The `acc` and `val_acc` normally differ from each other due to different split sizes.
  
    + Try same experiment with `validation_split=0.01` and `validation_split=0.4` and you will see how both `accuracy` and `val_acc` will change.
    + Normally the greater the validation split, the more similar both metrics will be since the validation split will be big enough to be representative (let's say it has cats and dogs, not only cats), taking into account that you need enough data to train correctly. This explains why in some cases the `val_acc` is higher than `accuracy` and vice versa.
  * Overfitting only occurs when the graph fashion or tendency changes and `val_acc` starts to drop and `accuracy` keeping increasing. This means that your model can not do any better with the validation dataset (non previously seen images).
  
  I work with `loss` and `val_loss` which are highly correlated with `accuracy`. Normally the loss is the inverse, so interpret the comments above in the inverse sense (sorry about the confusion but I'm taking this example from my current experiments) I hope it helps:
  
  [![enter image description here](https://i.sstatic.net/zvku1.png)](https://i.sstatic.net/zvku1.png)
  
  There are 2 experiments, orange and grey.
  
  * In both experiments, `val_loss` is always slightly higher than `loss` (because of my current validation split which it happens to be also 0.2, but normally is 0.01 and val\_loss is even higher).
  * On both experiments the `loss` trend is linearly decreasing, this is because gradient descent works and the loss functions is well defined and it converges.
  * Orange experiment is overfitting from epoch 20 onwards because, the `val_loss` won't drop any more and, on the contrary, it start increasing.
  * Grey experiment is just right, both `loss` and `val_loss` are still decreasing, and although the `val_loss` might be greater than `loss` it is not overfitting because it is still decreasing. So that is why it is still training :)
  
  Complex concepts here, I hope I was able to explain myself clearly! Cheers
  
  > **Comments:**
  > **username_0**: Thank you for the detailed explanation! Just a few questions: 1) If loss is computed after each batch and `val_acc` is computed only at the end of one epoch, does `val_acc`/`accuracy` reflect the accuracy of the entire model or only the batch? 2) Acc vs loss: when one is increasing, does the other necessarily have to decrease proportionally? Thank you  
  > **username_1**: 1) In Keras after computing the `accuracy` for the batch, it is averaged with the accuracy from all the previous batches, so yes, it takes them all into account and hence it is representative for the accuracy of your model. Same with `val_acc` (but with only one batch) 2) In 99% of cases yes, but it depends on how you define loss and accuracy (in standard keras metrics they are inversely proportional)  
  > **username_1**: I edited the response to add this  
  > **username_2**: Also note that using `mode.evaluate` to the training data may give you a different loss than what is printed during `model.fit`, since `fit` calculates the loss after each epoch separately and the model's parameters are being updated in-between.  
  
  ---
  
  