CUDA operations are called asynchronously in PyTorch.
If you would like to time certain operations, make sure to synchronize before starting and stopping the timer:
torch.cuda.synchronize()
t0 = time.time()
# CUDA op
torch.cuda.synchronize()
t1 = time.time()

Currently your code might accumulate other timings in a host to device copy and thus yield wrong timing results.