What do you mean with the point is to skip using a scheduler?
I would since it’s diverging in your case still try and lower the learning rate below the default value and see if that changes anything.
Edit: if that doesn’t help then I think some code would be helpful to further debug