You could create checkpoints (i.e. storing the state_dicts of the model, optimizer, etc.) and try to narrow down the iteration causing the issue. Assuming restoring the training routine works, you could then start the training from a few iterations before the invalid values are created and could then enable anomaly detection.