This might be the case and you could adapt the patience value for it.
E.g. if you current training sees a lot of these stats, you might want to use a higher patience to avoid decreasing the learning rate too early.