Could you check, if your loss is creating a NaN output?
If so, are you seeing this NaN also if you disable apex/amp?
Also, we recommend to try out the PyTorch native amp support, which is available in the nightly binaries as torch.cuda.amp 5.