Manually finding a sweet spot for the learning rate might be tricky and I believe that FastAI has written a “learning rate finder”, which tries to find a learning rate yielding the largest loss decrease, so you might want to search for their implementation.