Thank you for the reference! I just tried torch.backends.cudnn.benchmark =True and also remove unecessary .item() from the code, and the results (I only run it 1x):
Paperspace Paid P6000, Cuda 11.4:

with item(); cudnn.benchmark=False : ~164s
without item(); cudnn.benchmark=True : ~152s

Colab Free K80, Cuda 11.1:

with item(); cudnn.benchmark=False : ~110s
without item(); cudnn.benchmark=True : ~85s

I also try to run it on RTX A6000 from JarvisLab.ai:

without item(); cudnn.benchmark=True: ~111s

K80 still outperforms P6000 and A6000