I meant to say that it would be much faster if the multi-worker version worked properly.
In the first snippet, GPU utilization is very low. The call to .to(device) starts only after the current batch being processed by the GPU is done which causes a lot of GPU idle time. The same implementation runs much faster in TensorFlow.