I’m using torch.utils.data.DataLoader with a custom data.Dataset for loading video batches.
Since video is quiet heavy, I’m experiencing some latency issues, especially with the first batches of every epoch. For example:
2020-09-14 00:16:42,229 Epoch 120/200      Iteration 0/322    Loss: 9.586519445292652e-05    Time: 22.14 seconds/iteration
2020-09-14 00:18:24,911 Epoch 120/200      Iteration 50/322    Loss: 0.00014619871217291802    Time: 2.447 seconds/iteration
2020-09-14 00:19:35,139 Epoch 120/200      Iteration 100/322    Loss: 9.589765249984339e-05    Time: 1.931 seconds/iteration
2020-09-14 00:21:04,289 Epoch 120/200      Iteration 150/322    Loss: 8.543662988813594e-05    Time: 1.882 seconds/iteration
2020-09-14 00:22:16,219 Epoch 120/200      Iteration 200/322    Loss: 9.707502613309771e-05    Time: 1.772 seconds/iteration
2020-09-14 00:23:53,514 Epoch 120/200      Iteration 250/322    Loss: 0.00016338478599209338    Time: 1.806 seconds/iteration
2020-09-14 00:25:03,165 Epoch 120/200      Iteration 300/322    Loss: 0.00013553885219153017    Time: 1.738 seconds/iteration

Notice how long are the first iterations of the training of this epoch, and it actually happens in every epoch. I assume it is because the iter method just resets the loader for each epoch, so I was wondering if there’s any solution for warming it up? I already maximized num_workers by setting it to 16.