I ran into some issues with occasionally getting a NaN during the forward pass which leads to my loss function becoming NaN as well and breaking the training process. Since the graphs I’m generating are quite complex, it’s difficult for me to pin down just what in the forward pass causes the NaNs.
I tried using torch.autograd.detect_anomaly but it only raises an error on the loss function which is unhelpful in my case. Is there a way to also get the errors directly in the forward pass so I can find the actual source of the issue?