I’m not sure it’s a bug.
Could you try to time all different parts of your training separately and see which one takes most of the time?
Note, that CUDA calls are asynchronous so you should synchronize before starting and stopping the timer using torch.cuda.synchronize.