There are a few ideas that tries to do test runs to determine the learning rate. I know of one that the fastai 12 library uses where they start with a low learning rate and increases that until the loss diverges. At the point where it diverges, the learning rate was to high so they dial it back down a bit.
That is easy to implement yourself in vanilla pytorch with one of the learning rate schedulers. If you tried a smaller learning rate and it gets the same result there might be other things that you can spend your time on to improve