The code looks alright.



 victorc25:

I’m checking the isnan() and isinf() for the loss and output right after the loss.


Note that a NaN output or loss would be a different issue than a valid output or loss and a constantly reduced scaling factor (and thus also the debugging steps). Which case are you dealing with?