Solved by ptrblck in post #2 
              
              
                What output does detect_anomaly yield? 
Were you able to isolate the NaN to a few (or a single) iteration? 
If so, you could use forward hooks and store temporarily each submodules output in order to track down the source of the first NaN occurrence.