You could try to run your code with torch.autograd.detect_anomaly and check, if a backward pass fails.
If you cannot isolate the issue and think that the forward pass could create invalid outputs, you could use forward hooks to check the outputs of all layers and narrow down the issue.