What output does detect_anomaly yield?
Were you able to isolate the NaN to a few (or a single) iteration?
If so, you could use forward hooks and store temporarily each submodules output in order to track down the source of the first NaN occurrence.