I’m not sure to understand the question properly.
It seems your model is learning at the beginning but stops learning in further epochs.
If that’s the case, it’s a common observation.
You could try to change some hyperparameters, e.g. lower the learning rate when no learning takes place.
Have a look at lr_scheduler.ReduceLROnPlateau 5 to automatically lower the learning rate once a plateau is reached.