Since the weight and bias are almost perfectly fit, the loss value should be quite small at the end of the training. If you lower the learning rate further (to 1e-8), the parameter updates will be very small and might even get lost due to floating point precision.
You could try to increase the learning rate a bit or skip some reductions and see, if that would lower the loss. However, since your model is almost perfectly fit, I’m not sure if it’s worth a lot of effort to get the “perfect” numbers.