Thanks for the update and the great debugging!
Just a follow up question, since I’ve seen the updates just now:



 tueboesen:

The result is that suddenly the model returns nans even though all weights in the model appear reasonable.


How does this fit into your previous findings, i.e. neither the model output, nor the parameters or the gradients were having invalid values, but the optimizer.step() caused the parameters to become NaNs?
Before I saw the other posts I was trying to reason about this issue, but wasn’t sure how this could be happening.