How did you time your code?
Note that CUDA calls are asynchronous, so that you would have to synchronize your code before starting and stopping your timer using:
torch.cuda.synchronize()
t0 = time.perf_counter()
# Your code
torch.cuda.synchronize()
t1 = time.perf_counter()

Also, the first CUDA call shouldn’t be timed, as it might have some overhead due to context creation.