Maybe you’re overfitting but the underlying relationships are simple so your validation set still has decent accuracy but higher loss.
I feel like the change in accuracy could be caused by shuffling. Are you shuffling your data during training but not on test data? Does order matter for your problem?