your benchmarking is likely wrong.
Make sure you do such benchmarking for GPUs:
torch.cuda.synchronize() # finish all previous cuda calls
tm = time.time()
out = model(input)
torch.cuda.synchronize() # finish all cuda calls
print(time.time() - tm)