I suggest to let it run some more while the trend is still going down and see what happens. Probably the model is not able to generalise well beyond that point. Perhaps you need larger model? You might also look how well balanced your classes are, perhaps add some more training data if possible (or augmentation), increase batch size etc.