Suggestions:

normally you should put the scheduler step after every epochs and not every batch. This may cause your learning rate to get really small very fast, practically causing the model the learn nothing.
Maybe the pretrained Bert model has some parameter that freezes its parameters, and you need to specifically allow it to train?