Hi:)
Maybe you should share some information more as training parameters such as lr, epochs, optimizer, dropout.
Are you using model BERT from huggingface?
Which are the input features?
Try to expand a bit the question in order to let us better understand 
Anyway I will investigate the learning rate. If it is to high the model do not converge, try to lower it and add some dropout and weights normalization.