This description sounds as if your model is just not properly learning to generalize.
If the model is overfitting (low training loss, high validation loss), try to add regularization, change the model architecture, add more data augmentation etc.