I solved my problem using a custom dataloader developed by DKFZ group, https://github.com/MIC-DKFZ/batchgenerators 63. They have their own version of multi-threaded dataloader developed by Fabian Isensee. The timings I now get are more reasonable and “consistent”:
batch size=32 and 60 iterations/epoch and 10 epochs
Epoch 0: 10.64
Epoch 1: 10.19
Epoch 2: 10.12
Epoch 3: 10.19
Epoch 4: 11.27
Epoch 5: 10.87
Epoch 6: 10.48
Epoch 7: 10.3
Epoch 8: 10.23
Epoch 9: 10.48
Same thing with 5 epochs and 300 iteration/epoch
Epoch 0: 51.95
Epoch 1: 50.35
Epoch 2: 51.69
Epoch 3: 50.38
Epoch 4: 51.71