ericlormul:

Yeah, deg always contains zero elements. This explains why torch.autograd.set_detect_anomaly(True) triggers warning, but this warning shouldn’t be considered as an error, right?


When you enable anomaly detection, the NaNs will trigger an error. If you are not concerned about it, you could disable anomaly detection and handle the NaN values separately.
The idea of using this utility is to get a runtime error in order to debug the issue.



 ericlormul:

What confuses me is for the first ~600 iteration, there is no error. Then, nan grad appears, causing certain parameters to be updated to nan. Thus, in next iteration, due to nan parameters, pred tensor contains nan and bce_loss errors. Any insights on how to debug this. Thanks.


I guess that the first 600 iterations don’t contain an exact zero and would thus still create valid gradients. To void the NaN gradient, you could add a small eps value to the pow operation.