Well, as I said, I’d try that first. I’ve definitely run into situations where LR was preventing my networks from learning further. If that is indeed the case in your situation, nothing else you do will improve the loss because you just keep bouncing around the local minimum but never approach it.
As another trick, I’d start with SGD before moving on to Adam. In my own applications Adam converges faster but SGD still generalizes better in the end (by a few percent).