<p>very interesting questions:</p>
<p><strong>1. what exactly is happening when training and validation accuracy change during training</strong></p>
<ul>
<li>The <code>accuracy</code> change after every batch computation. You have 588 batches, so loss will be computed after each one of these batches (let's say each batch have 8 images). However, the accuracy you see in the progress bar it is the accuracy of the current batch <em>averaged</em> with the accuracy of all the previous batches so far. See <code>keras.utils.generic_utils.Progbar</code>.</li>
<li>The <code>val_acc</code> is computed only at the end of one epoch and it is computed with all your validation dataset at once (considering it as a single batch, so if you have 100 images for validation it will compute accuracy as a single batch of 100 images)</li>
</ul>
<p><strong>2. what do different behaviours imply</strong></p>
<ul>
<li><p>The <code>acc</code> and <code>val_acc</code> normally differ from each other due to different split sizes.</p>
<ul>
<li>Try same experiment with <code>validation_split=0.01</code> and <code>validation_split=0.4</code> and you will see how both <code>accuracy</code> and <code>val_acc</code> will change.</li>
<li>Normally the greater the validation split, the more similar both metrics will be since the validation split will be big enough to be representative (let's say it has cats and dogs, not only cats), taking into account that you need enough data to train correctly. This explains why in some cases the <code>val_acc</code> is higher than <code>accuracy</code> and vice versa.</li>
</ul>
</li>
<li><p>Overfitting only occurs when the graph fashion or tendency changes and <code>val_acc</code> starts to drop and <code>accuracy</code> keeping increasing. This means that your model can not do any better with the validation dataset (non previously seen images).</p>
</li>
</ul>
<p>I work with <code>loss</code> and <code>val_loss</code> which are highly correlated with <code>accuracy</code>. Normally the loss is the inverse, so interpret the comments above in the inverse sense (sorry about the confusion but I'm taking this example from my current experiments) I hope it helps:</p>
<p><a href="https://i.sstatic.net/zvku1.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zvku1.png" alt="enter image description here" /></a></p>
<p>There are 2 experiments, orange and grey.</p>
<ul>
<li><p>In both experiments, <code>val_loss</code> is always slightly higher than <code>loss</code> (because of my current validation split which it happens to be also 0.2, but normally is 0.01 and val_loss is even higher).</p>
</li>
<li><p>On both experiments the <code>loss</code> trend is linearly decreasing, this is because gradient descent works and the loss functions is well defined and it converges.</p>
</li>
<li><p>Orange experiment is overfitting from epoch 20 onwards because, the <code>val_loss</code> won't drop any more and, on the contrary, it start increasing.</p>
</li>
<li><p>Grey experiment is just right, both <code>loss</code> and <code>val_loss</code> are still decreasing, and although the <code>val_loss</code> might be greater than <code>loss</code> it is not overfitting because it is still decreasing. So that is why it is still training :)</p>
</li>
</ul>
<p>Complex concepts here, I hope I was able to explain myself clearly! Cheers</p>

CC BY-SA 4.0
2019-12-04T12:21:10.897Z
Thank you for the detailed explanation! Just a few questions: 1) If loss is computed after each batch and `val_acc` is computed only at the end of one epoch, does `val_acc`/`accuracy` reflect the accuracy of the entire model or only the batch? 2) Acc vs loss: when one is increasing, does the other necessarily have to decrease proportionally? Thank you
nilsinelabore
CC BY-SA 4.0
2019-12-04T13:33:22.827Z
1) In Keras after computing the `accuracy` for the batch, it is averaged with the accuracy from all the previous batches, so yes, it takes them all into account and hence it is representative for the accuracy of your model. Same with `val_acc` (but with only one batch)
2) In 99% of cases yes, but it depends on how you define loss and accuracy (in standard keras metrics they are inversely proportional)
JVGD
CC BY-SA 4.0
2019-12-04T13:41:57.663Z
I edited the response to add this
JVGD
CC BY-SA 4.0
2022-05-23T10:14:51.977Z
Also note that using `mode.evaluate` to the training data may give you a different loss than what is printed during `model.fit`, since `fit` calculates the loss after each epoch separately and the model's parameters are being updated in-between.
NikoNyrh
CC BY-SA 4.0
2019-12-04T10:18:26.693Z
2022-05-22T21:42:33.110Z
2022-05-22T21:42:33.110Z
JVGD
<p>During model training, I noticed various behaviour in between training and validation accuracy. I understand that <a href="https://stackoverflow.com/a/51345413/11901732">'The training set is used to train the model, while the validation set is only used to evaluate the model's performance...'</a>, but I'd like to know if there is any relationship between training and validation accuracy and, if yes,</p>
<ol>
<li><p><strong>what exactly is happening when training and validation accuracy change during training and;</strong></p>
</li>
<li><p><strong>what do different behaviours imply</strong></p>
</li>
</ol>
<p>For instance, some believe there is overfitting problem if training &gt; validation accuracy. What happens if one is greater than the other <strong>alternately</strong>, which is the case below?</p>
<p>Here is the code</p>
<pre><code>inputs_1 = keras.Input(shape=(10081,1))

layer1 = Conv1D(64,14)(inputs_1)
layer2 = layers.MaxPool1D(5)(layer1)
layer3 = Conv1D(64, 14)(layer2)
layer4 = layers.GlobalMaxPooling1D()(layer3)

inputs_2 = keras.Input(shape=(104,))             
layer5 = layers.concatenate([layer4, inputs_2])
layer6 = Dense(128, activation='relu')(layer5)
layer7 = Dense(2, activation='softmax')(layer6)


model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], output = [layer7])
model_2.summary()


X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:10185], df[['Result_cat','Result_cat1']].values, test_size=0.2) 
X_train = X_train.to_numpy()
X_train = X_train.reshape([X_train.shape[0], X_train.shape[1], 1]) 
X_train_1 = X_train[:,0:10081,:]
X_train_2 = X_train[:,10081:10185,:].reshape(736,104)   


X_test = X_test.to_numpy()
X_test = X_test.reshape([X_test.shape[0], X_test.shape[1], 1]) 
X_test_1 = X_test[:,0:10081,:]
X_test_2 = X_test[:,10081:10185,:].reshape(185,104)    

adam = keras.optimizers.Adam(lr = 0.0005)
model_2.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['acc'])

history = model_2.fit([X_train_1,X_train_2], y_train, epochs = 120, batch_size = 256, validation_split = 0.2, callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)])
</code></pre>
<p>model summary</p>
<pre><code>/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:15: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=[&lt;tf.Tenso..., outputs=[&lt;tf.Tenso...)`
  from ipykernel import kernelapp as app
Model: &quot;model_3&quot;
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            (None, 10081, 1)     0                                            
__________________________________________________________________________________________________
conv1d_5 (Conv1D)               (None, 10068, 64)    960         input_5[0][0]                    
__________________________________________________________________________________________________
max_pooling1d_3 (MaxPooling1D)  (None, 2013, 64)     0           conv1d_5[0][0]                   
__________________________________________________________________________________________________
conv1d_6 (Conv1D)               (None, 2000, 64)     57408       max_pooling1d_3[0][0]            
__________________________________________________________________________________________________
global_max_pooling1d_3 (GlobalM (None, 64)           0           conv1d_6[0][0]                   
__________________________________________________________________________________________________
input_6 (InputLayer)            (None, 104)          0                                            
__________________________________________________________________________________________________
concatenate_3 (Concatenate)     (None, 168)          0           global_max_pooling1d_3[0][0]     
                                                                 input_6[0][0]                    
__________________________________________________________________________________________________
dense_5 (Dense)                 (None, 128)          21632       concatenate_3[0][0]              
__________________________________________________________________________________________________
dense_6 (Dense)                 (None, 2)            258         dense_5[0][0]                    
==================================================================================================
Total params: 80,258
Trainable params: 80,258
Non-trainable params: 0
</code></pre>
<p>and the training process</p>
<pre><code>__________________________________________________________________________________________________
Train on 588 samples, validate on 148 samples
Epoch 1/120
588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
Epoch 2/120
588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
Epoch 3/120
588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
Epoch 4/120
588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
Epoch 5/120
588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
Epoch 6/120
588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
Epoch 7/120
588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
Epoch 8/120
588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
Epoch 9/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
Epoch 10/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
Epoch 11/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
Epoch 12/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
Epoch 13/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
Epoch 14/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
Epoch 15/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
Epoch 16/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
Epoch 17/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
Epoch 18/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
Epoch 19/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
Epoch 20/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
Epoch 21/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
Epoch 22/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
Epoch 23/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
Epoch 24/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
Epoch 25/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
Epoch 26/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
Epoch 27/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
Epoch 28/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
Epoch 29/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
Epoch 30/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
Epoch 31/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
Epoch 32/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
Epoch 33/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
Epoch 34/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
Epoch 35/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
Epoch 36/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
Epoch 37/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
Epoch 38/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
Epoch 39/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
Epoch 40/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
Epoch 41/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
Epoch 42/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
Epoch 43/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
Epoch 44/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
Epoch 45/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
Epoch 46/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
Epoch 47/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
Epoch 48/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
Epoch 49/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
Epoch 50/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
Epoch 51/120
588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
Epoch 52/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
Epoch 53/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
Epoch 54/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
Epoch 55/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
Epoch 56/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
Epoch 57/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
Epoch 58/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
Epoch 59/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
Epoch 60/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
Epoch 61/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
Epoch 62/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
Epoch 63/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
Epoch 64/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
Epoch 65/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
Epoch 66/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
Epoch 67/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
Epoch 68/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
Epoch 69/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
Epoch 70/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
Epoch 71/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041__________________________________________________________________________________________________
Train on 588 samples, validate on 148 samples
Epoch 1/120
588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
Epoch 2/120
588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
Epoch 3/120
588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
Epoch 4/120
588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
Epoch 5/120
588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
Epoch 6/120
588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
Epoch 7/120
588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
Epoch 8/120
588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
Epoch 9/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
Epoch 10/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
Epoch 11/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
Epoch 12/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
Epoch 13/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
Epoch 14/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
Epoch 15/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
Epoch 16/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
Epoch 17/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
Epoch 18/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
Epoch 19/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
Epoch 20/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
Epoch 21/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
Epoch 22/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
Epoch 23/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
Epoch 24/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
Epoch 25/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
Epoch 26/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
Epoch 27/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
Epoch 28/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
Epoch 29/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
Epoch 30/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
Epoch 31/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
Epoch 32/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
Epoch 33/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
Epoch 34/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
Epoch 35/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
Epoch 36/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
Epoch 37/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
Epoch 38/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
Epoch 39/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
Epoch 40/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
Epoch 41/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
Epoch 42/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
Epoch 43/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
Epoch 44/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
Epoch 45/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
Epoch 46/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
Epoch 47/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
Epoch 48/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
Epoch 49/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
Epoch 50/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
Epoch 51/120
588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
Epoch 52/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
Epoch 53/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
Epoch 54/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
Epoch 55/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
Epoch 56/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
Epoch 57/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
Epoch 58/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
Epoch 59/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
Epoch 60/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
Epoch 61/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
Epoch 62/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
Epoch 63/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
Epoch 64/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
Epoch 65/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
Epoch 66/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
Epoch 67/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
Epoch 68/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
Epoch 69/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
Epoch 70/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
Epoch 71/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041
</code></pre>
<p>Notice how at first <code>acc</code> is lower than <code>val_acc</code> and later is greater than <code>val_acc</code>. Can someone please shed some light what could be happening here? Thank you</p>

<neural-networks><machine-learning><training><accuracy><cross-validation>
What is the relationship between the training accuracy and validation accuracy?
nilsinelabore
# What is the relationship between the training accuracy and validation accuracy?

(Asked by: username_0 on 2019-12-03)

During model training, I noticed various behaviour in between training and validation accuracy. I understand that 'The training set is used to train the model, while the validation set is only used to evaluate the model's performance...', but I'd like to know if there is any relationship between training and validation accuracy and, if yes,

what exactly is happening when training and validation accuracy change during training and;

what do different behaviours imply


For instance, some believe there is overfitting problem if training > validation accuracy. What happens if one is greater than the other alternately, which is the case below?
Here is the code
inputs_1 = keras.Input(shape=(10081,1))

layer1 = Conv1D(64,14)(inputs_1)
layer2 = layers.MaxPool1D(5)(layer1)
layer3 = Conv1D(64, 14)(layer2)
layer4 = layers.GlobalMaxPooling1D()(layer3)

inputs_2 = keras.Input(shape=(104,))             
layer5 = layers.concatenate([layer4, inputs_2])
layer6 = Dense(128, activation='relu')(layer5)
layer7 = Dense(2, activation='softmax')(layer6)


model_2 = keras.models.Model(inputs = [inputs_1, inputs_2], output = [layer7])
model_2.summary()


X_train, X_test, y_train, y_test = train_test_split(df.iloc[:,0:10185], df[['Result_cat','Result_cat1']].values, test_size=0.2) 
X_train = X_train.to_numpy()
X_train = X_train.reshape([X_train.shape[0], X_train.shape[1], 1]) 
X_train_1 = X_train[:,0:10081,:]
X_train_2 = X_train[:,10081:10185,:].reshape(736,104)   


X_test = X_test.to_numpy()
X_test = X_test.reshape([X_test.shape[0], X_test.shape[1], 1]) 
X_test_1 = X_test[:,0:10081,:]
X_test_2 = X_test[:,10081:10185,:].reshape(185,104)    

adam = keras.optimizers.Adam(lr = 0.0005)
model_2.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['acc'])

history = model_2.fit([X_train_1,X_train_2], y_train, epochs = 120, batch_size = 256, validation_split = 0.2, callbacks = [keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)])

model summary
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:15: UserWarning: Update your `Model` call to the Keras 2 API: `Model(inputs=[
and the training process
__________________________________________________________________________________________________
Train on 588 samples, validate on 148 samples
Epoch 1/120
588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
Epoch 2/120
588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
Epoch 3/120
588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
Epoch 4/120
588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
Epoch 5/120
588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
Epoch 6/120
588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
Epoch 7/120
588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
Epoch 8/120
588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
Epoch 9/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
Epoch 10/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
Epoch 11/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
Epoch 12/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
Epoch 13/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
Epoch 14/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
Epoch 15/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
Epoch 16/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
Epoch 17/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
Epoch 18/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
Epoch 19/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
Epoch 20/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
Epoch 21/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
Epoch 22/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
Epoch 23/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
Epoch 24/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
Epoch 25/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
Epoch 26/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
Epoch 27/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
Epoch 28/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
Epoch 29/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
Epoch 30/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
Epoch 31/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
Epoch 32/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
Epoch 33/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
Epoch 34/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
Epoch 35/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
Epoch 36/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
Epoch 37/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
Epoch 38/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
Epoch 39/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
Epoch 40/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
Epoch 41/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
Epoch 42/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
Epoch 43/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
Epoch 44/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
Epoch 45/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
Epoch 46/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
Epoch 47/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
Epoch 48/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
Epoch 49/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
Epoch 50/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
Epoch 51/120
588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
Epoch 52/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
Epoch 53/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
Epoch 54/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
Epoch 55/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
Epoch 56/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
Epoch 57/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
Epoch 58/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
Epoch 59/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
Epoch 60/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
Epoch 61/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
Epoch 62/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
Epoch 63/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
Epoch 64/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
Epoch 65/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
Epoch 66/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
Epoch 67/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
Epoch 68/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
Epoch 69/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
Epoch 70/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
Epoch 71/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041__________________________________________________________________________________________________
Train on 588 samples, validate on 148 samples
Epoch 1/120
588/588 [==============================] - 16s 26ms/step - loss: 5.6355 - acc: 0.4932 - val_loss: 4.1086 - val_acc: 0.6216
Epoch 2/120
588/588 [==============================] - 15s 25ms/step - loss: 4.5977 - acc: 0.5748 - val_loss: 3.8252 - val_acc: 0.4459
Epoch 3/120
588/588 [==============================] - 15s 25ms/step - loss: 4.3815 - acc: 0.4575 - val_loss: 2.4087 - val_acc: 0.6622
Epoch 4/120
588/588 [==============================] - 15s 25ms/step - loss: 3.7480 - acc: 0.6003 - val_loss: 2.0060 - val_acc: 0.6892
Epoch 5/120
588/588 [==============================] - 15s 25ms/step - loss: 3.3019 - acc: 0.5408 - val_loss: 2.3176 - val_acc: 0.5676
Epoch 6/120
588/588 [==============================] - 15s 25ms/step - loss: 3.1739 - acc: 0.5663 - val_loss: 2.2607 - val_acc: 0.6892
Epoch 7/120
588/588 [==============================] - 15s 25ms/step - loss: 3.2322 - acc: 0.6207 - val_loss: 1.8898 - val_acc: 0.7230
Epoch 8/120
588/588 [==============================] - 15s 25ms/step - loss: 2.9777 - acc: 0.6020 - val_loss: 1.8401 - val_acc: 0.7500
Epoch 9/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8982 - acc: 0.6429 - val_loss: 1.8517 - val_acc: 0.7365
Epoch 10/120
588/588 [==============================] - 15s 25ms/step - loss: 2.8342 - acc: 0.6344 - val_loss: 1.7941 - val_acc: 0.7095
Epoch 11/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7426 - acc: 0.6327 - val_loss: 1.8495 - val_acc: 0.7162
Epoch 12/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7340 - acc: 0.6531 - val_loss: 1.7652 - val_acc: 0.7162
Epoch 13/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6680 - acc: 0.6616 - val_loss: 1.8097 - val_acc: 0.7365
Epoch 14/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6922 - acc: 0.6786 - val_loss: 1.7143 - val_acc: 0.7500
Epoch 15/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6161 - acc: 0.6786 - val_loss: 1.6960 - val_acc: 0.7568
Epoch 16/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6054 - acc: 0.6905 - val_loss: 1.6779 - val_acc: 0.7297
Epoch 17/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6072 - acc: 0.6684 - val_loss: 1.6750 - val_acc: 0.7703
Epoch 18/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5907 - acc: 0.6871 - val_loss: 1.6774 - val_acc: 0.7432
Epoch 19/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5854 - acc: 0.6718 - val_loss: 1.6609 - val_acc: 0.7770
Epoch 20/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5621 - acc: 0.6905 - val_loss: 1.6709 - val_acc: 0.7365
Epoch 21/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5515 - acc: 0.6854 - val_loss: 1.6904 - val_acc: 0.7703
Epoch 22/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5749 - acc: 0.6837 - val_loss: 1.6862 - val_acc: 0.7297
Epoch 23/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6212 - acc: 0.6514 - val_loss: 1.7215 - val_acc: 0.7568
Epoch 24/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6532 - acc: 0.6633 - val_loss: 1.7105 - val_acc: 0.7230
Epoch 25/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7300 - acc: 0.6344 - val_loss: 1.6870 - val_acc: 0.7432
Epoch 26/120
588/588 [==============================] - 15s 25ms/step - loss: 2.7355 - acc: 0.6650 - val_loss: 1.6733 - val_acc: 0.7703
Epoch 27/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6336 - acc: 0.6650 - val_loss: 1.6572 - val_acc: 0.7297
Epoch 28/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6018 - acc: 0.6803 - val_loss: 1.7292 - val_acc: 0.7635
Epoch 29/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5448 - acc: 0.7143 - val_loss: 1.8065 - val_acc: 0.7095
Epoch 30/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5724 - acc: 0.6820 - val_loss: 1.8029 - val_acc: 0.7297
Epoch 31/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6622 - acc: 0.6650 - val_loss: 1.6594 - val_acc: 0.7568
Epoch 32/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6211 - acc: 0.6582 - val_loss: 1.6375 - val_acc: 0.7770
Epoch 33/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5911 - acc: 0.6854 - val_loss: 1.6964 - val_acc: 0.7500
Epoch 34/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5050 - acc: 0.7262 - val_loss: 1.8496 - val_acc: 0.6892
Epoch 35/120
588/588 [==============================] - 15s 25ms/step - loss: 2.6012 - acc: 0.6752 - val_loss: 1.7443 - val_acc: 0.7432
Epoch 36/120
588/588 [==============================] - 15s 25ms/step - loss: 2.5688 - acc: 0.6871 - val_loss: 1.6220 - val_acc: 0.7568
Epoch 37/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4843 - acc: 0.7279 - val_loss: 1.6166 - val_acc: 0.7905
Epoch 38/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4707 - acc: 0.7449 - val_loss: 1.6496 - val_acc: 0.7905
Epoch 39/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4683 - acc: 0.7109 - val_loss: 1.6641 - val_acc: 0.7432
Epoch 40/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4671 - acc: 0.7279 - val_loss: 1.6553 - val_acc: 0.7703
Epoch 41/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4479 - acc: 0.7347 - val_loss: 1.6302 - val_acc: 0.7973
Epoch 42/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4355 - acc: 0.7551 - val_loss: 1.6241 - val_acc: 0.7973
Epoch 43/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4286 - acc: 0.7568 - val_loss: 1.6249 - val_acc: 0.7973
Epoch 44/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4250 - acc: 0.7585 - val_loss: 1.6248 - val_acc: 0.7770
Epoch 45/120
588/588 [==============================] - 14s 25ms/step - loss: 2.4198 - acc: 0.7517 - val_loss: 1.6212 - val_acc: 0.7703
Epoch 46/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4246 - acc: 0.7568 - val_loss: 1.6129 - val_acc: 0.7838
Epoch 47/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4237 - acc: 0.7517 - val_loss: 1.6166 - val_acc: 0.7973
Epoch 48/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4287 - acc: 0.7432 - val_loss: 1.6309 - val_acc: 0.8041
Epoch 49/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4179 - acc: 0.7381 - val_loss: 1.6271 - val_acc: 0.7838
Epoch 50/120
588/588 [==============================] - 15s 25ms/step - loss: 2.4164 - acc: 0.7381 - val_loss: 1.6258 - val_acc: 0.7973
Epoch 51/120
588/588 [==============================] - 14s 24ms/step - loss: 2.1996 - acc: 0.7398 - val_loss: 1.3612 - val_acc: 0.7973
Epoch 52/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1387 - acc: 0.8265 - val_loss: 1.4811 - val_acc: 0.7973
Epoch 53/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1607 - acc: 0.8078 - val_loss: 1.5060 - val_acc: 0.7838
Epoch 54/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1783 - acc: 0.8129 - val_loss: 1.4878 - val_acc: 0.8176
Epoch 55/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1745 - acc: 0.8197 - val_loss: 1.4762 - val_acc: 0.8108
Epoch 56/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1764 - acc: 0.8129 - val_loss: 1.4631 - val_acc: 0.7905
Epoch 57/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1637 - acc: 0.8078 - val_loss: 1.4615 - val_acc: 0.7770
Epoch 58/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1563 - acc: 0.8112 - val_loss: 1.4487 - val_acc: 0.7703
Epoch 59/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1396 - acc: 0.8146 - val_loss: 1.4362 - val_acc: 0.7905
Epoch 60/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1240 - acc: 0.8316 - val_loss: 1.4333 - val_acc: 0.8041
Epoch 61/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1173 - acc: 0.8333 - val_loss: 1.4369 - val_acc: 0.8041
Epoch 62/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1228 - acc: 0.8384 - val_loss: 1.4393 - val_acc: 0.8041
Epoch 63/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1113 - acc: 0.8316 - val_loss: 1.4380 - val_acc: 0.8041
Epoch 64/120
588/588 [==============================] - 15s 25ms/step - loss: 1.1102 - acc: 0.8452 - val_loss: 1.4217 - val_acc: 0.8041
Epoch 65/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0961 - acc: 0.8469 - val_loss: 1.4129 - val_acc: 0.7973
Epoch 66/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0903 - acc: 0.8537 - val_loss: 1.4019 - val_acc: 0.8041
Epoch 67/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0890 - acc: 0.8503 - val_loss: 1.3850 - val_acc: 0.8176
Epoch 68/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0878 - acc: 0.8520 - val_loss: 1.4035 - val_acc: 0.7635
Epoch 69/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0984 - acc: 0.8469 - val_loss: 1.4060 - val_acc: 0.8041
Epoch 70/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0893 - acc: 0.8418 - val_loss: 1.3981 - val_acc: 0.7973
Epoch 71/120
588/588 [==============================] - 15s 25ms/step - loss: 1.0876 - acc: 0.8605 - val_loss: 1.3951 - val_acc: 0.8041

Notice how at first acc is lower than val_acc and later is greater than val_acc. Can someone please shed some light what could be happening here? Thank you

**Tags:** neural-networks, machine-learning, training, accuracy, cross-validation


---

## 1 answers

### Answer by username_1 on 2019-12-04. Score: 1 | Accepted answer

very interesting questions:

**1. what exactly is happening when training and validation accuracy change during training**

* The `accuracy` change after every batch computation. You have 588 batches, so loss will be computed after each one of these batches (let's say each batch have 8 images). However, the accuracy you see in the progress bar it is the accuracy of the current batch *averaged* with the accuracy of all the previous batches so far. See `keras.utils.generic_utils.Progbar`.
* The `val_acc` is computed only at the end of one epoch and it is computed with all your validation dataset at once (considering it as a single batch, so if you have 100 images for validation it will compute accuracy as a single batch of 100 images)

**2. what do different behaviours imply**

* The `acc` and `val_acc` normally differ from each other due to different split sizes.

  + Try same experiment with `validation_split=0.01` and `validation_split=0.4` and you will see how both `accuracy` and `val_acc` will change.
  + Normally the greater the validation split, the more similar both metrics will be since the validation split will be big enough to be representative (let's say it has cats and dogs, not only cats), taking into account that you need enough data to train correctly. This explains why in some cases the `val_acc` is higher than `accuracy` and vice versa.
* Overfitting only occurs when the graph fashion or tendency changes and `val_acc` starts to drop and `accuracy` keeping increasing. This means that your model can not do any better with the validation dataset (non previously seen images).

I work with `loss` and `val_loss` which are highly correlated with `accuracy`. Normally the loss is the inverse, so interpret the comments above in the inverse sense (sorry about the confusion but I'm taking this example from my current experiments) I hope it helps:

[![enter image description here](https://i.sstatic.net/zvku1.png)](https://i.sstatic.net/zvku1.png)

There are 2 experiments, orange and grey.

* In both experiments, `val_loss` is always slightly higher than `loss` (because of my current validation split which it happens to be also 0.2, but normally is 0.01 and val\_loss is even higher).
* On both experiments the `loss` trend is linearly decreasing, this is because gradient descent works and the loss functions is well defined and it converges.
* Orange experiment is overfitting from epoch 20 onwards because, the `val_loss` won't drop any more and, on the contrary, it start increasing.
* Grey experiment is just right, both `loss` and `val_loss` are still decreasing, and although the `val_loss` might be greater than `loss` it is not overfitting because it is still decreasing. So that is why it is still training :)

Complex concepts here, I hope I was able to explain myself clearly! Cheers

> **Comments:**
> **username_0**: Thank you for the detailed explanation! Just a few questions: 1) If loss is computed after each batch and `val_acc` is computed only at the end of one epoch, does `val_acc`/`accuracy` reflect the accuracy of the entire model or only the batch? 2) Acc vs loss: when one is increasing, does the other necessarily have to decrease proportionally? Thank you  
> **username_1**: 1) In Keras after computing the `accuracy` for the batch, it is averaged with the accuracy from all the previous batches, so yes, it takes them all into account and hence it is representative for the accuracy of your model. Same with `val_acc` (but with only one batch) 2) In 99% of cases yes, but it depends on how you define loss and accuracy (in standard keras metrics they are inversely proportional)  
> **username_1**: I edited the response to add this  
> **username_2**: Also note that using `mode.evaluate` to the training data may give you a different loss than what is printed during `model.fit`, since `fit` calculates the loss after each epoch separately and the model's parameters are being updated in-between.  

---

