Thanks,
I have downloaded the latest PyTorch with CUDA 10.2 using (pip install torch torchvision). To reduce possible errors, I tested it in a machine with CUDA 10.2 as well. Unfortunately, the training time remains slow when using torch.cuda.amp.
System:
Ubuntu 18.04
Cuda: 10.2
Pytorch: 1.6
GPU: 2 GTX 1080Ti