I am training a face recognition model with DDP on 8 GPUs, but I am getting loss Nan occasionally. The issue does not happen every time, but at a very high frequency, e.g. 8 out of 10 times, and the occurring time differ from each other (which makes it harder to debug).
The pipeline is: images are first fed into a DDP model for feature extraction, then those features are gathered and input into different classifiers on different GPUs to get the results and finally compute losses.
After debugging, I find that whenever the loss is nan for a certain GPU, the features extracted by this GPU is Nan. I checked the inputs, they are within normal ranges of [-1, +1]. And according to my understanding, the DDP models should be always the same among all the GPUs, it’s weird only one of the GPU outputs Nan while the rest are normal.
What might be the issue? I’ve been checking it for several days but still cannot find the problem.
-----------------------update---------------------
I debugged with torch.autograd.set_detect_anomaly, and the traceback shows ‘MmBackward’ returned nan values in its 0th output:
2020-11-02 23_14_04-10.80.42.177 (jiali)1870×483 33.1 KB
2020-11-02 23_14_49-10.80.42.177 (jiali)1636×323 20.3 KB