Solved by vfdev-5 in post #4 
              
              
                I’ve just tested your code to train resnet18 QAT 8 bits on CIFAR10 with DP on 2 GPUs and I also have nans with loss and y_pred. Then tested on a single gpu without DP and the model looks like learning something. 
Maybe, you can try to wrap the model with torch DDP such that you could use all 8 GPUs: …