It’s expected behavior that your GPU can’t process anything while waiting for the next batch, since your code runs in the main thread. Multiple workers should help in this case, but as you explained it’s apparently not possible to use them due to some system restrictions. I’m still unsure, why you would have to kill the processes manually.
Could you try to remove the training code and just time the data loading?
Basically just the time to get the next batch without any processing on the GPU etc.