Thanks for the reply!
I checked the inputs for F.linear, weights is normal and there is no Nan values, but emb_all has Nan. Since emb_all is the concatenation of embs from all 8 GPUs, I am very confused why it happens. All the 8 GPUs should share exactly the same weights of model (under DDP mode), if the input images are normal, then the output embs for all GPUs should also be normal.
The loss value is normal as well, e.g. 40 and I set a small lr, so it is not divergence. There is no 0 div, sqrt 0, or log 0 operations. I’ll double check this anyway.
BTW, is there any other reasons for Mmbackward returning nan? I cannot find many info on this, but overflow is mentioned in this blog: https://github.com/NVIDIA/apex/issues/440 5