I tried very small learning rates and exponents like 0.0001,0.9999,1.0001. It’s only nan if I choose a different value than 0 or 1. I also checked out the gradients after the first backward pass. Therefore I don’t think the loss is exploding.