Are you running your code on the CPU or GPU?
In the latter case, note that CUDA calls are asynchronous, so that you would need to synchronize the code before starting and stopping the timer:
torch.cuda.synchronize()
t0 = time.perf_counter()
# your code
torch.cuda.synchronize()
t1 = time.perf_counter()