In your training loop there is still some pre-processing / creation of data going on.
Since this will be called sequentially, you’ll most likely see your GPU waiting for it.
Could you try to time the noise and pose generation?
Also, why did you set num_workers to zero? This would mean that the main thread is now responsible to load the data which might also slow down your code.