In that case your model might just be overfitting.
Does the validation loss go down at the beginning of your training and stays after a certain amount of epochs constant or goes up again?
You might need to play around with some hyperparameters or add regularization (e.g. via dropout).