@andreaskoepf Thanks for the reply, yup with a lr of 0.01 it works fine but what throws me off is this insane sensitivity of the optimizer. If I switch to adam I don’t get nan but I get stagnation so loss is ~same.



 andreaskoepf:

You could test whether gradually warming up the learning rate is beneficial.


What do we mean these days by warming up the learning rate, there are so many heuristics that’s its hard to keep track of them all.