If your predictions are better and the loss is higher, while the output shape is equal, your model might not assign high or low logits to the predictions even though they are right.
Are you able to continue the training and lower the loss using your attention model?