That is the correct way to manually change a learning rate and it’s fine to use it with Adam. As for the reason your loss increases when you change it. We can’t even guess without knowing how you’re changing the learning rate (increase or decrease), if that’s the training or validation loss/accuracy, and details about the problem you’re solving. The reasons could be anything from “you’re choosing the wrong learning rate” to “Your optimization jumped out of a local minimum”.
It’s likely best to get more intuition as to what’s happening with the optimization on your own if you’re interested.
adam paper 179
sgdr paper 93