Your scheduler is reducing the learning rate such that the training seems to get stalled.
You could remove the scheduler and check if your model is able to learn the train set. Once this is possible, you could then try to make sure it is also able to generalize well to the validation dataset.