I am currently training a model where the loss explodes if I use a higher learning rate. So the highest learning rate I can use is like 1e-3. The loss even goes to NaN after the first iteration, which was a bit surprising to me.
I thought I had some bug but the model does converge after a long time and I get meaningful predictions from it.
I am currently not using any regularization and no weight decay.
I was wondering if there is some advise on what I could try and do to push the learning rate up.