Yes, even with torch.backends.cudnn.benchmark=True placed at the beginning of the script it is slow using a GTX1080Ti. On the other hand, It is around 3 days of training using torch.cuda.amp with a pair of RTX2080Ti’s without any problems. This reinforces more my theory about how torch.cuda.amp is optimized.