I think the loss.item() op creates a synchronization point so that your script would have to wait for the CUDA kernels to finish their execution.
Apparently if you call it directly after loss.backward() some kernels cannot run in background and you see a worse performance.