First, note that your scales are different on each graph so this can be quite misleading.
Accuracy values are not so different if you take a closer look.
Many problems can happen during non-convex optimization of deep networks.
One problem could be that a smaller learning rate makes it more likely to get stuck in a local minima. Thus your model could be less general.
Please have a look on this link 18. Pasting relevant part below:

Low learning rates not only slow down training: they can even degrade the performance of the model. Though this guideline is still controversial, current conventional wisdom says that large learning rates increase generalization ability.
There are several explanations to this. One is that larger learning rates increase the noise on the stochastic gradient, which acts as an implicit regularizer. Another related explanation is that larger learning rates cause the model to converge to " wider " minima that generalize better. This is more fully explored in this paper 3, but the key idea is that “sharper” minima (minima where the loss function changes greatly when the parameters change slightly) generalize worse than minima where the loss function does not change as largely. This makes intuitive sense: for instance, if you were training an image classifier, you would expect inputs corresponding to similar images to have similar outputs, and thus similar loss values. If the loss changes rapidly, you wouldn’t expect the model to be robust to unseen inputs.