Hi,
All the CUDA API is asynchronous so the backward call will return before everything is actually computed.
Your last line actually access some data from the GPU and thus forces a synchronization and thus waits for all computation to finish.
You can add before your last line a torch.cuda.synchronize() and your last line will become instant again.