without more information it’s hard to decipher where the issue lies, a few thoughts of what may be causing this:

training set may feature noisy samples that make convergence in training a tough task
test set may feature samples that are not representative of the noise in the training set

Is the model showing progress in regards to the loss and metric you are using while training?