I have no answer, but I saw this issue is not alone. Another Embedding-Bidirectional-LSTM showed the same effect: https://www.kaggle.com/c/stanford-covid-vaccine/discussion/186485 12
I hope someone can detect why that is.