Yes, sure! I’ll be able to share something on Monday.
I’m starting to believe my code (which was actually developed 1.5 years ago) might not be optimal for the current version of Pytorch. I might be accumulating something somewhere that with the latest updates causes slowdowns, I’m don’t know.
A BIG update though: with torch==1.9.0+cu111 on the server with CUDA 11.0 if I set the following:
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = True # in the past this was False
torch.backends.cudnn.enabled = False # does it make sense to disable after setting benchmark True?

The speed is normal again