Since you are trying to time CUDA operations, you should synchronize before starting and stopping your timer using torch.cuda.synchronize().
torch.cuda.synchronize()
st = time.time()
...
torch.cuda.synchronize()
print(time.time() - st)

Currently you are just seeing the time to launch the kernels.