Additional information:
My first training epoch on a small dataset takes ~90 seconds. The dataloader loop (regardless of training or for validation), with the same batchsize runs significantly slower. The validation takes 180 seconds and the second training epoch (onward) takes 2~3 times the time the first epoch took. And this remains more or less constant. On a large dataset, this is more pronounced.
What is noticeable that in the first epoch the GPU is hardly idle and from the 2nd epoch (onward) it is idle almost all the time. I don’t know what this is related to (hardware issue or code). I am at my wits end to and hoping someone can help diagnose the issue.
As a test, I stripped the code of all training and just created batches and skipped thorough rest of the operations except applying . I noticed that for a batch size=32 and 60 iterations/epoch and 10 epochs. I noted the following timings:
Epoch 0: 15.64
Epoch 1: 61.19
Epoch 2: 59.28
Epoch 3: 63.12
Epoch 4: 62.46
Epoch 5: 60.43
Epoch 6: 60.72
Epoch 7: 59.05
Epoch 8: 64.78
Epoch 9: 62.96
Same thing with 5 epochs and 300 iteration/epoch
Epoch 0: 61.79
Epoch 1: 253.94
Epoch 2: 255.45
Epoch 3: 255.12
Epoch 4: 255.57
So the problem is definitely in the data loading part. I should mention that my data is being loaded using a custom dataset from a video. I am not sure why it should affect the performance from the 2nd epoch onward.