Hi,
I guess that you’re running things on the GPU? And you do not use torch.cuda.synchronize() before measuring time?
Keep in mind that the CUDA API is asynchronous. So if you don’t sync, you only measure the time to ask the GPU to do stuff. But not how long it took to actually do it. In this case, by the time you get to the backward, the async queue is most likely full and you start waiting for computations to actually happen.